AI 助理的工作架構要怎麼修改,才不會花很多 token,還能做好程式任務?
- 這篇 Google 論文讓最佳化工具修改固定不變的 Gemini 3.8 Flash 周邊架構,並比較三種做法。在 SWE-bench Lite 上,整份重寫的答對率是 42.5%,每項任務用了 451.6k tokens。
- 一次只改一個部分、通過保留題目檢查才採用的做法,答對率是 40.0%,每項任務用了 46.7k tokens。這種做法也用在 Gemini 4 Argon,沒有再搜尋修改方式,答對率為 37.5%,高於整份重寫的 30.0%。測試只有 40 題,數字只能看出初步方向。
- 論文也提醒,自我修改可能造成提示詞膨脹、鑽評測漏洞或動手腳改評分。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
這項研究比較三種修改 AI 助理工作架構的方法,呈現逐一檢查修改如何兼顧任務表現與 token 用量。
Another very interesting paper from Google on self-improving agent harnesses.
(bookmark it)
They let an optimizer rewrite the harness around a frozen Gemini 3.8 Flash on SWE-bench Lite and compared three approaches.
- A prompt rule file lowered held-out accuracy from 5.0% to 2.5%.
- A full rewrite of the harness reached 42.5% at 451.6k tokens per task.
- Gated edits to one module at a time reached 40.0% at 46.7k tokens per task.
The gated version splits the harness into four parts (context, control, tools, and verification). Each round it patches one part, checks the patch on held-out tasks under a token cap, and deploys it only if it passes.
The gated harness also transferred to Gemini 4 Argon with no new search, solving 37.5% against 30.0% for the full rewrite. The test set is small (40 held-out issues), so treat the numbers as directional.
The paper also lists failure modes to guard against when a harness edits itself, including prompt bloat, benchmark gaming, and evaluator tampering.
Chat with Paper: https://t.co/Ns1EHBjrxi