現有程式解題測驗越來越難分出模型實力,因為許多 AI 助理已經能解開大部分題目。
- Google 的 BuggyMLE 讓 AI 助理在 222 個可運作的機器學習專案裡,故意加入可控制的程式錯誤,再用這些專案做出更難的修復任務。測試中,14 個模型裡表現最好的模型,解題率從最簡單難度的 95% 降到最難的 9%。
- 經過錯誤任務訓練的 Qwen3.5-9B,則在這項測試中逼近頂尖模型,並把 SWE-bench Lite 解題率從 24% 提高到 33%。這種做法能用既有專案製作更有挑戰的測驗與訓練資料。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
Banger paper from Google on ML engineering agents.
Pay attention to this one, builders. It's a clever way to scale ML engineering by injecting bugs.
(bookmark it)
Repository-level coding benchmarks are saturating. BuggyMLE makes tasks harder on purpose by having agents inject controlled bugs into 222 working ML repositories.
- A Code Complexity Index built from ten repo features predicts average solve rates across eight coding benchmarks with R^2 = 0.951.
- On the graded MLE-Bench Lite variant, the best of 14 models drops from a 95% solve rate on the easiest tier to 9% on the hardest.
- A Qwen3.5-9B trained on the bug curriculum rivals frontier models there and raises SWE-bench Lite from 24% to 33%.
Patch size and code complexity are the strongest predictors of difficulty, so the generator can raise both to produce harder tasks and training data from the same repos.
This is a great example of how to build harder tasks from your own repos to improve the overall system.
Chat with Paper: https://t.co/xtU0wkytOH