跳到正文
𝕏 Artificial Analysis(x.com)·· 9 天前精選品質評分61
重點導讀

用 AI 助理寫程式時,分數高不一定代表不同工具在同一套測試裡都能公平比較。

  • Artificial Analysis 更新 Coding Agent Index,改用新版 FrontierCode 和 Terminal-Bench 評分。
  • 前者看程式修改是否值得專案維護者合併,後者更新了檢查答案的方式,讓評分更可靠,也較不容易被鑽漏洞。新版中 Claude Fable 5.2 和 Claude Opus 5.5 都拿下 58 分。
  • 整體分數平均比舊版低 5 分。得分 48 分以上的配置,每項任務成本約差 30 倍,從 $1.51 到 $44.22,顯示比較表現時也要留意花費。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

推薦理由

新版基準讓模型分數平均下降 5 分,並列出不同配置的單項任務成本,方便比較分數與花費之間的差異。

Artificial Analysis
Artificial Analysis@ArtificialAnlys
·9 天前
𝕏

Announcing Artificial Analysis Coding Agent Index v2.1, upgrading FrontierCode and Terminal-Bench. Claude Fable 5.2 and Claude Opus 5.5 share the lead at 58

The Coding Agent Index measures models in agent harnesses; v2.1 of the Index upgrades FrontierCode from 1.1 to 2.0 and Terminal-Bench from 4.0 to 4.1.

Upgraded:

➤ FrontierCode-AA 2.0 (replacing FrontierCode v1.1): our implementation of Cognition's FrontierCode 2.0, comprising 132 real-world coding tasks graded on whether a repository maintainer would merge the agent's change

➤ Terminal-Bench 4.1 (replacing Terminal-Bench 4.0): the same 66 tasks, scored with updated verifiers that improve reliability and resistance to reward hacking

Staying in the Index: SWE-Atlas-QnA-AA

Key results:

➤ Claude Fable 5.2 and Claude Opus 5.5 share the lead at 58: Claude Fable 5.2 scores 58 at both xhigh and high in Claude Code. Claude Opus 5.5 (max) scores 58 in Claude Code and, paired with GPT-6 Luna High, in Cognition's multi-model harness Devin Fusion CLI. Claude Fable 5.2 (max) scores 56 as more of its attempts hit task time limits. Claude Sonnet 5.5 (max, 56) in Claude Code and Gemini 4 Carbon (high, 55) in Antigravity CLI follow

➤ Index scores are lower in v2.1, driven by the upgrade to FrontierCode-AA 2.0: models score lower by 5 points on average. Codex with GPT-6.1 Terra (max) is the only configuration which didn’t fall between versions, sitting within 1 point of its prior score

➤ Cost per task varies ~30x among configurations scoring 48+: from $1.51 for Codex with GPT-6.1 Sol (max, 52) to $44.22 for Claude Code with Claude Fable 5.2 (max, 56)

推文來源 · X (Twitter)在 X 參與討論

讀者討論

理性交流 · 即時討論