讓 AI 助理修改自己的程式框架時,整份重寫可能很耗 token,也可能改壞原本能做好的事。
- Google 研究讓最佳化模型一次只改 4 個部分中的 1 個,並用保留的任務和 token 上限檢查修改,不合格就退回。在 40 項 SWE-bench Lite 任務中,逐項檢查的版本解出 40.0%,每項任務用 46.7k tokens。
- 未經檢查的整份重寫解出 42.5%,卻用了 451.6k tokens。結果顯示,逐次檢查修改能用較少 token 換得接近的準確率。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
New Google paper finds that an unchecked full rewrite of an agent harness matched accuracy but cost nearly 10 times more tokens per task than gated modular edits.
that self-improving agent harnesses work best when they change 1 code module at a time and must pass validation.
The harness is the code around a model that manages context, runs tools, and checks work before stopping. A harness tuned for 1 model often stalls or wastes tokens on another.
They let an optimizer model improve a 142-line coding harness by editing 1 of its 4 parts at a time. Each edit had to pass a validation set and a token cap, or it was rejected.
On 40 held-out SWE-bench Lite tasks with Gemini 3.8 Flash, the gated harness solved 40.0% against 5.0% for the seed, using 46.7k tokens per task. A full unchecked rewrite solved 42.5% but used 451.6k tokens, and prompt rules dropped accuracy to 2.5%.
If you let agents tune their harness, gate each change on held-out tasks and token cost, and enforce rules in code, not prompts.