跳到正文
𝕏 elvis(x.com)·· 2 小時前品質評分49
重點導讀

現有程式解題測驗越來越難分出模型實力,因為許多 AI 助理已經能解開大部分題目。

  • Google 的 BuggyMLE 讓 AI 助理在 222 個可運作的機器學習專案裡,故意加入可控制的程式錯誤,再用這些專案做出更難的修復任務。測試中,14 個模型裡表現最好的模型,解題率從最簡單難度的 95% 降到最難的 9%。
  • 經過錯誤任務訓練的 Qwen3.5-9B,則在這項測試中逼近頂尖模型,並把 SWE-bench Lite 解題率從 24% 提高到 33%。這種做法能用既有專案製作更有挑戰的測驗與訓練資料。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

elvis
elvis@omarsar0
·2 小時前
𝕏

Banger paper from Google on ML engineering agents.

Pay attention to this one, builders. It's a clever way to scale ML engineering by injecting bugs.

(bookmark it)

Repository-level coding benchmarks are saturating. BuggyMLE makes tasks harder on purpose by having agents inject controlled bugs into 222 working ML repositories.

- A Code Complexity Index built from ten repo features predicts average solve rates across eight coding benchmarks with R^2 = 0.951.
- On the graded MLE-Bench Lite variant, the best of 14 models drops from a 95% solve rate on the easiest tier to 9% on the hardest.
- A Qwen3.5-9B trained on the bug curriculum rivals frontier models there and raises SWE-bench Lite from 24% to 33%.

Patch size and code complexity are the strongest predictors of difficulty, so the generator can raise both to produce harder tasks and training data from the same repos.

This is a great example of how to build harder tasks from your own repos to improve the overall system.

https://t.co/Qy6Jrd1jmp

Chat with Paper: https://t.co/xtU0wkytOH

推文來源 · X (Twitter)在 X 參與討論
來源: elvis (x.com)
在 X 查看推文

讀者討論

理性交流 · 即時討論