跳到正文
𝕏 Rohan Paul(x.com)·· 3 天前品質評分57
重點導讀

讓 AI 助理修改自己的程式框架時,整份重寫可能很耗 token,也可能改壞原本能做好的事。

  • Google 研究讓最佳化模型一次只改 4 個部分中的 1 個,並用保留的任務和 token 上限檢查修改,不合格就退回。在 40 項 SWE-bench Lite 任務中,逐項檢查的版本解出 40.0%,每項任務用 46.7k tokens。
  • 未經檢查的整份重寫解出 42.5%,卻用了 451.6k tokens。結果顯示,逐次檢查修改能用較少 token 換得接近的準確率。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

Rohan Paul
Rohan Paul@rohanpaul_ai
·3 天前
𝕏

New Google paper finds that an unchecked full rewrite of an agent harness matched accuracy but cost nearly 10 times more tokens per task than gated modular edits.

that self-improving agent harnesses work best when they change 1 code module at a time and must pass validation.

The harness is the code around a model that manages context, runs tools, and checks work before stopping. A harness tuned for 1 model often stalls or wastes tokens on another.

They let an optimizer model improve a 142-line coding harness by editing 1 of its 4 parts at a time. Each edit had to pass a validation set and a token cap, or it was rejected.

On 40 held-out SWE-bench Lite tasks with Gemini 3.8 Flash, the gated harness solved 40.0% against 5.0% for the seed, using 46.7k tokens per task. A full unchecked rewrite solved 42.5% but used 451.6k tokens, and prompt rules dropped accuracy to 2.5%.

If you let agents tune their harness, gate each change on held-out tasks and token cost, and enforce rules in code, not prompts.

推文來源 · X (Twitter)在 X 參與討論

讀者討論

理性交流 · 即時討論