跳到正文
𝕏 elvis(x.com)·· 2 天前品質評分63
重點導讀

好的程式評測不容易製作,逐題請人標記答案也很花工夫。

  • Meta 的研究提出 Benchmark Factory,讓 AI 助理從已合併的內部程式碼變更中挑選任務、寫出題目,再用不同強度的模型檢查評測結果,反覆修正直到通過品質檢查。
  • 這些檢查不需要每題都有人標記答案。研究者也故意拿太簡單或評分方式有問題的評測來測試,系統能把它們擋下。用未參與建置的設定測試時,較高運算強度的 Claude Opus 4.7 排名高於較低強度,GPT-5.5 也高於 GPT-5.4。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

elvis
elvis@omarsar0
·2 天前
𝕏

Interesting paper from Meta showing how to build coding evals from your own codebase.

(bookmark it)

Good benchmarks are hard to create, but using this method with agents looks promising.

They propose Benchmark Factory, an agent loop that turns merged internal code changes into benchmarks. It writes the query that selects tasks, runs models of known relative strength, and repairs the benchmark until it passes a set of quality checks. These checks need no per-task human labels.

To make sure the checks catch bad benchmarks, the authors fed them bad benchmarks on purpose. A benchmark that was far too easy got rejected. So did benchmarks with five different kinds of broken grading. When they scrambled which model was which, a bad benchmark slipped through at most 1.4% of the time.

On configurations never used to build them, the benchmarks ranked Claude Opus 4.7 at high effort above low effort (60.6% vs 54.4%) and GPT-5.5 above GPT-5.4 (65.8% vs 60.5%). 9 of 10 per-team comparisons agreed.

https://t.co/LyNVrLwKiv

Chat with Paper: https://t.co/sg5STh6Z6r

推文來源 · X (Twitter)在 X 參與討論
來源: elvis (x.com)
在 X 查看推文

讀者討論

理性交流 · 即時討論