Amazon 的研究顯示,在使用者事實回憶任務中,透過 on-policy 自我蒸餾寫入權重的模型,在不提供檢索上下文時可勝過文字檢索基線。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
Weight memory vs. retrieval in agents. Brilliant work from Amazon.
This work shows weight memory can outperform text memory on recall of user facts.
But the best part is that you can leverage both.
Using the same model and user data, this work shows that plain fine-tuning stores user facts in weights at 36.9% accuracy, while on-policy self-distillation reaches 59.9%.
That second number also beats retrieval.
On eight HaluMem-Long users, the distilled model answers with no retrieved context and beats MemOS-style retrieval (47.8%), extracted context (51.9%), and the raw dialogue (51.3%). The gain holds for every user and at 4B, 9B, and 27B.
How it works: a teacher, which is the same model reading the user's memory, trains a student that has to answer without it.
Weight memory has clear limits. It forgets older facts and cannot tell when something was never recorded. A verify-then-regenerate router checks each answer against the text store and adds 6.2 points.
The recommendation is to keep a text store for recent and missing facts, and move stable user knowledge into weights.
Chat with Paper: https://t.co/mT9PXpK7WJ