跳到正文
𝕏 Rohan Paul(x.com)·· 14 天前品質評分63
重點導讀

摘要 Amazon 研究指出,許多 AI 稅務、法律與商業規則基準在評分時因答案鍵或評分程式錯誤,將正確答案標記為錯誤。

  • 調查範圍:審核 9 個公開基準(RuleArena、LegalBench 等),重跑參考程式並測試評分器。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

Rohan Paul
Rohan Paul@rohanpaul_ai
·14 天前
𝕏

New Amazon paper finds that benchmarks used to grade AI models on tax, legal and business rules often mark correct answers wrong, because the tests’ own answer keys or grading code contain mistakes.

Benchmarks like RuleArena and LegalBench test whether models apply written rules to tax, legal, and business cases. If the reference answer or the scorer is wrong, the score measures the benchmark's bugs, not the model.

They audited 9 public benchmarks by tracing references back to the source rules, rerunning reference programs, and testing scorers with controlled answers. Then they rescored the same saved model responses after each fix.

LegalBench's tax scorer deleted decimal points, so $259,487.13 was read as 25,948,713. Fixing it raised accepted answers from 8 or 9 to between 87 and 91 out of 96 for each of 3 models.

Before trusting a low score on a rule-following benchmark, read the scorer code and check a few failed answers by hand.

推文來源 · X (Twitter)在 X 參與討論

讀者討論

理性交流 · 即時討論