跳到正文
𝕏 elvis(x.com)·· 16 天前品質評分56
重點導讀

Amazon 的研究顯示,在使用者事實回憶任務中,透過 on-policy 自我蒸餾寫入權重的模型,在不提供檢索上下文時可勝過文字檢索基線。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

elvis
elvis@omarsar0
·16 天前
𝕏

Weight memory vs. retrieval in agents. Brilliant work from Amazon.

This work shows weight memory can outperform text memory on recall of user facts.

But the best part is that you can leverage both.

Using the same model and user data, this work shows that plain fine-tuning stores user facts in weights at 36.9% accuracy, while on-policy self-distillation reaches 59.9%.

That second number also beats retrieval.

On eight HaluMem-Long users, the distilled model answers with no retrieved context and beats MemOS-style retrieval (47.8%), extracted context (51.9%), and the raw dialogue (51.3%). The gain holds for every user and at 4B, 9B, and 27B.

How it works: a teacher, which is the same model reading the user's memory, trains a student that has to answer without it.

Weight memory has clear limits. It forgets older facts and cannot tell when something was never recorded. A verify-then-regenerate router checks each answer against the text store and adds 6.2 points.

The recommendation is to keep a text store for recent and missing facts, and move stable user knowledge into weights.

https://t.co/PC1iuyOh9B

Chat with Paper: https://t.co/mT9PXpK7WJ

推文來源 · X (Twitter)在 X 參與討論
來源: elvis (x.com)
在 X 查看推文

讀者討論

理性交流 · 即時討論