跳到正文
𝕏 elvis(x.com)·· 4 天前精選品質評分71
重點導讀

AI 助理的工作架構要怎麼修改,才不會花很多 token,還能做好程式任務?

  • 這篇 Google 論文讓最佳化工具修改固定不變的 Gemini 3.8 Flash 周邊架構,並比較三種做法。在 SWE-bench Lite 上,整份重寫的答對率是 42.5%,每項任務用了 451.6k tokens。
  • 一次只改一個部分、通過保留題目檢查才採用的做法,答對率是 40.0%,每項任務用了 46.7k tokens。這種做法也用在 Gemini 4 Argon,沒有再搜尋修改方式,答對率為 37.5%,高於整份重寫的 30.0%。測試只有 40 題,數字只能看出初步方向。
  • 論文也提醒,自我修改可能造成提示詞膨脹、鑽評測漏洞或動手腳改評分。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

推薦理由

這項研究比較三種修改 AI 助理工作架構的方法,呈現逐一檢查修改如何兼顧任務表現與 token 用量。

elvis
elvis@omarsar0
·4 天前
𝕏

Another very interesting paper from Google on self-improving agent harnesses.

(bookmark it)

They let an optimizer rewrite the harness around a frozen Gemini 3.8 Flash on SWE-bench Lite and compared three approaches.

- A prompt rule file lowered held-out accuracy from 5.0% to 2.5%.
- A full rewrite of the harness reached 42.5% at 451.6k tokens per task.
- Gated edits to one module at a time reached 40.0% at 46.7k tokens per task.

The gated version splits the harness into four parts (context, control, tools, and verification). Each round it patches one part, checks the patch on held-out tasks under a token cap, and deploys it only if it passes.

The gated harness also transferred to Gemini 4 Argon with no new search, solving 37.5% against 30.0% for the full rewrite. The test set is small (40 held-out issues), so treat the numbers as directional.

The paper also lists failure modes to guard against when a harness edits itself, including prompt bloat, benchmark gaming, and evaluator tampering.

https://t.co/RmzvB7ewRK

Chat with Paper: https://t.co/Ns1EHBjrxi

推文來源 · X (Twitter)在 X 參與討論
來源: elvis (x.com)
在 X 查看推文

讀者討論

理性交流 · 即時討論