跳到正文
hackernews100(simple-bench.com)· algoth1·· 1 天前品質評分52

SimpleBench 排行榜顯示 GPT-6.2 Astra、Sol 分數低於人類基準

Astra 6.2 and Sol 6.2 flop at Simple Bench

重點導讀

AI 模型在日常推理題上不一定比一般人答得好。

  • SimpleBench 收錄超過 200 題選擇題,涵蓋空間與時間推理、社交理解和容易讓人答錯的文字陷阱。
  • 網站列出的 9 位參與者人類基準是 83.7%。排行榜上,GPT-6.2 Astra 得分 77.3%,GPT-6.2 Sol 得分 69.9%,都低於這個基準。這些結果反映的是這套測驗中的表現,不能直接代表模型在所有任務上的能力。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

正文

SimpleBench

SimpleBench

Where Everyday Human Reasoning ~~Still Surpasses~~ Frontier Models

SimpleBench Team

Latest LeaderboardReportTry YourselfImage 1: Hugging FacePublic DatasetCode

Introduction

We introduce SimpleBench, a multiple-choice text benchmark for LLMs where individuals with unspecialized (high school) knowledge outperform SOTA models. SimpleBench includes over 200 questions covering spatio-temporal reasoning, social intelligence, and what we call linguistic adversarial robustness (or trick questions). For the vast majority of text-based benchmarks LLMs outperform a non-specialized human, and increasingly, exceed expert human performance. However, on SimpleBench, a non-specialized human baseline is 83.7%, based on our small sample of nine participants, ~~outperforming every tested LLM, including today's top model, Claude Fable, which scored 81.9%~~. While we expect model performance to improve over time, the results of SimpleBench confirm that the memorized knowledge, and approximate reasoning retrieval, utilized by frontier LLMs is not always enough to answer basic questions just yet.

For a deeper dive into our results and our methods, check out the full technical report here.

Use all of these models on the Simple Bench app - LMcouncil.ai

Leaderboard

MCQ Perf/$Timeline Open-Ended

RankModelScore (AVG@5)Organization
-Highest Human Score*95.4%
1st NEWClaude Fable 5.294.9%Anthropic
2nd NEWGemini 4 Neon94.4%Google
3rd NEWGemini 4 Carbon92.6%Google
4th NEWGemini 4 Argon91.7%Google
5th NEWGemini 4 Hydrogen91.7%Google
6thClaude Opus 5.588.4%Anthropic
7thClaude Fable 5.186.6%Anthropic
8thGPT-6 Astra Pro86.5%OpenAI
-Human Baseline*83.7%
9thGPT-6 Astra83.6%OpenAI
10thGPT-6.1 Sol82.9%OpenAI
11thGemini 3.8 Flash82.4%Google
12thClaude Fable81.9%Anthropic
13thMuse Spark 1.381.8%Meta
14thClaude Opus 580.6%Anthropic
15thGemini 3.1 Pro Preview79.6%Google
16th NEWGPT-6.2 Astra Pro79.2%OpenAI
17th NEWGPT-6.2 Astra77.3%OpenAI
18thGPT-5.5 Pro76.9%OpenAI
19thGemini 3.5 Flash76.7%Google
20thGemini 3 Pro Preview76.4%Google
21stGrok 4.675.9%xAI
22ndClaude Sonnet 5.575.9%Anthropic
23rdMuse Spark 1.274.5%Meta
24thGPT-6 Sol73.1%OpenAI
25th NEWMuse Ember72.7%Meta
26th NEWGPT-6.1 Terra72.2%OpenAI
27thGPT-5.6 Sol Pro (xhigh)71.7%OpenAI
28thQwen 3.7 Max70.4%Alibaba
29thGrok 4.570.0%xAI
30th NEWGPT-6.2 Sol69.9%OpenAI
31stGPT-5.569.0%OpenAI
32ndClaude Opus 4.667.6%Anthropic
33rdDeepSeek V4.1 Flash66.7%DeepSeek
34thGLM 5.366.2%Z.ai
35thClaude Opus 4.864.8%Anthropic
36thGPT-5.6 Sol (xhigh)64.8%OpenAI
37thQwen 3.6 Max Preview63.0%Alibaba
38thQwen 3.8 2.4T A95B62.5%Alibaba
39thGemini 2.5 Pro (06-05)62.4%Google
40thClaude Opus 4.562.0%Anthropic
41stClaude Opus 4.761.7%Anthropic
42ndGPT-5 Pro61.6%OpenAI
43rdGemini 3 Flash Preview61.1%Google
44thDeepSeek V4 Flash61.1%DeepSeek
45thKimi K3 (max)60.7%Moonshot AI
46thClaude Sonnet 560.6%Anthropic
47thGrok 460.5%xAI
48thQwen 3.8 27B60.2%Alibaba
49thClaude 4.1 Opus60.0%Anthropic
50th NEWClaude Haiku 5.559.2%Anthropic
51stClaude 4 Opus58.8%Anthropic
52ndGLM 5.258.8%Z.ai
53rdKimi K2.7 Code57.9%Moonshot AI
54thGPT-5.2 Pro (xhigh)57.4%OpenAI
55thGPT-5 (high)56.7%OpenAI
56thGrok 4.1 Fast56.0%xAI
57thGLM 5.155.1%Z.ai
58thOx Alpha55.1%Unknown
59thClaude 4.5 Sonnet54.3%Anthropic
60thGPT-5.1 (high)53.2%OpenAI
61stGLM 553.2%Zhipu AI
62ndo3 (high)53.1%OpenAI
63rdDeepSeek 3.2 Speciale52.6%DeepSeek
64thGemini 2.5 Pro (03-25)51.6%Google
65thDeepSeek V4 Pro50.9%DeepSeek
66thInkling50.0%Thinking Machines
67thGPT-5.6 Terra (xhigh)48.9%OpenAI
68thGLM 4.747.7%Z.ai
69thKimi K2.546.8%Moonshot AI
70thGPT-5.6 Luna (xhigh)46.8%OpenAI
71stClaude 3.7 Sonnet (thinking)46.4%Anthropic
72ndDeepSeek V4 Flash (preview)46.3%DeepSeek
73rdGPT-5.2 (high)45.8%OpenAI
74thMiniMax M345.8%MiniMax
75thClaude 4 Sonnet (thinking)45.5%Anthropic
76thClaude 3.7 Sonnet44.9%Anthropic
77tho1-preview41.7%OpenAI
78thNemotron 3 Ultra41.7%Nvidia
79thClaude 3.5 Sonnet 10-2241.4%Anthropic
80thGemini 2.5 Flash (latest)41.2%Google
81stDeepSeek R1 05/2840.8%DeepSeek
82ndo1-2024-12-17 (high)40.1%OpenAI
83rdDeepSeek V3.140.0%DeepSeek
84thKimi K2 Thinking39.6%Moonshot AI
85tho4-mini (high)38.7%OpenAI
86tho1-2024-12-17 (med)36.7%OpenAI
87thGrok 336.1%xAI
88thQwen 3.6 Flash35.2%Alibaba
89thMiniMax M2.134.7%MiniMax
90thGPT-4.534.5%OpenAI
91stGemini-exp-120631.1%Google
92ndQwen3 235B-A22B31.0%Alibaba
93rdDeepSeek R130.9%DeepSeek
94thGemini 2.0 Flash Thinking30.7%Google
95thLlama 4 Maverick27.7%Meta
96thClaude 3.5 Sonnet 06-2027.5%Anthropic
97thDeepSeek V3 03-2427.2%DeepSeek
98thGemini 1.5 Pro 00227.1%Google
99thGPT-4.127.0%OpenAI
100thKimi K226.3%Moonshot AI
101stGPT-4 Turbo25.1%OpenAI
102ndMiniMax M225.0%MiniMax
103rdClaude 3 Opus23.5%Anthropic
104thLlama 3.1 405b instruct23.0%Meta
105tho3-mini (high)22.8%OpenAI
106thGrok 222.7%xAI
107thMistral Large v222.5%Mistral
108thGPT-OSS 120B22.1%OpenAI
109thMistral Large 320.4%Mistral
110thLlama 3.3 70b instruct19.9%Meta
111thDeepSeek V318.9%DeepSeek
112thGemini 2.0 Flash Exp18.9%Google
113tho1-mini18.1%OpenAI
114thGPT-4o 08-0617.8%OpenAI
115thCommand R+17.4%Cohere
116thGPT-4o mini10.7%OpenAI
117thGPT-3.5 Turbo8.0%OpenAI

benchmark settings

temperature: 0.7, top-p: 0.95 (except o1 series)

*See Human Evaluation section of Report for details on how we calculated Human Baseline.

**We try an engineered prompt to optimize benchmark specific performance. See LLM Eval section of Report for details.

Video Summary

Video 2

Go to the Patreon for the latest, fully-explained update

Please reach out to [aiexplained@outlook.com](mailto:aiexplained@outlook.com) for inquiries, business questions or feedback about SimpleBench. This page was was adopted from the Nerfies project page.

來源: hackernews100 (simple-bench.com)
開啟原文

讀者討論

理性交流 · 即時討論