自動改進 harness 時,如果換了模型或測試資料,原本有效的修改可能不再有用。
- Google 的研究提出一套記憶系統,會替每次修改記下原因、預期結果、實際結果和測試情境,讓系統在提出或挑選新修改前先參考。研究指出,這套方法在 4 種領域的跨資料集與 Opus 到 Gemini 測試中,都比只沿用原本的 harness 表現好。
- 10 次程式測試平均增加 2.36 分。研究也舉例,一套為 Opus 調整的 harness 換到 Gemini 後,分數低於未調整的版本。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
Banger paper from Google on RSI.
(bookmark it)
They propose an approach for continual recursive self-improvement at the harness level. They also propose their own self-improving memory system for the harness optimizer.
This clearly highlights the importance of traces and having a functional, adaptive, and structured persistent memory system in your harness.
They show that a harness optimizer improves 9 of 10 continued runs when it remembers why it made each past edit.
They discuss that a common problem shows up when you change the model or the benchmark. An Opus-evolved harness started at 68.41 on Gemini, below the 73.89 of the plain seed harness.
Self-Improvement Memory groups past edits by the harness problem they addressed. For each edit, it stores the rationale, predicted outcome, observed outcome, and evaluation context. The proposer and the selector both read it before making new changes.
Across four domains, including Terminal-Bench 2.1 to Terminal-Bench 4, SIM beats continuing from the harness alone in six cross-dataset and four Opus-to-Gemini settings. Across ten coding runs, it adds 2.36 points on average (p=0.003).
If you evolve harnesses automatically, store the reasons and results of each edit next to the code.
Chat with Paper: https://t.co/DcEJoYBDba