跳到正文
𝕏 Rohan Paul(x.com)·· 5 天前精選品質評分73
重點導讀

用大型語言模型替醫療答案打分時,如果評審偏愛自己寫的內容,整體比較結果就可能跟著偏斜。

  • Stanford 團隊讓 OpenAI、Anthropic、Google 和 Qwen 的 8 個模型回答 620 個臨床工作者提出的問題,再替 8 份匿名答案排序。每個模型給自己的答案排名都比其他 7 個評審更高,平均差 1.22 名次,GPT-5.6 Sol 接近 3 名。
  • 即使公開模型名稱,偏差也只減少 0.094 名次。研究建議使用不同模型家族組成評審團,並分別呈現各評審的結果。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

推薦理由

研究比較 8 個模型對匿名醫療答案的排序,呈現評審模型偏向自家答案的幅度,也說明多模型評審與分開呈現結果的做法。

Rohan Paul
Rohan Paul@rohanpaul_ai
·5 天前
𝕏

New Stanford paper finds that all 8 LLMs tested rank their own medical answers higher than other models rank them, even when answers are anonymous.

Medical AI benchmarks often use LLMs to grade other LLMs' answers. If a judge favors its own outputs, the benchmark tilts toward that model.

They had 8 models from OpenAI, Anthropic, Google and Qwen answer 620 real clinician questions, then rank all 8 anonymous answers. They compared where each judge placed its own answer with where the other 7 judges placed that same answer.

Every judge favored its own answer, by 1.22 rank positions on average and nearly 3 for GPT-5.6 Sol. The bias held without a rubric, and revealing model names cut it by only 0.094 positions.

If you use LLM judges, use a panel from several model families and report results per judge instead of 1 pooled score.

推文來源 · X (Twitter)在 X 參與討論

讀者討論

理性交流 · 即時討論