SimpleBench 排行榜顯示 GPT-6.2 Astra、Sol 分數低於人類基準
Astra 6.2 and Sol 6.2 flop at Simple Bench
AI 模型在日常推理題上不一定比一般人答得好。
- SimpleBench 收錄超過 200 題選擇題,涵蓋空間與時間推理、社交理解和容易讓人答錯的文字陷阱。
- 網站列出的 9 位參與者人類基準是 83.7%。排行榜上,GPT-6.2 Astra 得分 77.3%,GPT-6.2 Sol 得分 69.9%,都低於這個基準。這些結果反映的是這套測驗中的表現,不能直接代表模型在所有任務上的能力。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
SimpleBench
SimpleBench
Where Everyday Human Reasoning ~~Still Surpasses~~ Frontier Models
SimpleBench Team
Latest LeaderboardReportTry YourselfPublic DatasetCode
Introduction
We introduce SimpleBench, a multiple-choice text benchmark for LLMs where individuals with unspecialized (high school) knowledge outperform SOTA models. SimpleBench includes over 200 questions covering spatio-temporal reasoning, social intelligence, and what we call linguistic adversarial robustness (or trick questions). For the vast majority of text-based benchmarks LLMs outperform a non-specialized human, and increasingly, exceed expert human performance. However, on SimpleBench, a non-specialized human baseline is 83.7%, based on our small sample of nine participants, ~~outperforming every tested LLM, including today's top model, Claude Fable, which scored 81.9%~~. While we expect model performance to improve over time, the results of SimpleBench confirm that the memorized knowledge, and approximate reasoning retrieval, utilized by frontier LLMs is not always enough to answer basic questions just yet.
For a deeper dive into our results and our methods, check out the full technical report here.
Use all of these models on the Simple Bench app - LMcouncil.ai
Leaderboard
MCQ Perf/$Timeline Open-Ended
| Rank | Model | Score (AVG@5) | Organization |
|---|---|---|---|
| - | Highest Human Score* | 95.4% | |
| 1st NEW | Claude Fable 5.2 | 94.9% | Anthropic |
| 2nd NEW | Gemini 4 Neon | 94.4% | |
| 3rd NEW | Gemini 4 Carbon | 92.6% | |
| 4th NEW | Gemini 4 Argon | 91.7% | |
| 5th NEW | Gemini 4 Hydrogen | 91.7% | |
| 6th | Claude Opus 5.5 | 88.4% | Anthropic |
| 7th | Claude Fable 5.1 | 86.6% | Anthropic |
| 8th | GPT-6 Astra Pro | 86.5% | OpenAI |
| - | Human Baseline* | 83.7% | |
| 9th | GPT-6 Astra | 83.6% | OpenAI |
| 10th | GPT-6.1 Sol | 82.9% | OpenAI |
| 11th | Gemini 3.8 Flash | 82.4% | |
| 12th | Claude Fable | 81.9% | Anthropic |
| 13th | Muse Spark 1.3 | 81.8% | Meta |
| 14th | Claude Opus 5 | 80.6% | Anthropic |
| 15th | Gemini 3.1 Pro Preview | 79.6% | |
| 16th NEW | GPT-6.2 Astra Pro | 79.2% | OpenAI |
| 17th NEW | GPT-6.2 Astra | 77.3% | OpenAI |
| 18th | GPT-5.5 Pro | 76.9% | OpenAI |
| 19th | Gemini 3.5 Flash | 76.7% | |
| 20th | Gemini 3 Pro Preview | 76.4% | |
| 21st | Grok 4.6 | 75.9% | xAI |
| 22nd | Claude Sonnet 5.5 | 75.9% | Anthropic |
| 23rd | Muse Spark 1.2 | 74.5% | Meta |
| 24th | GPT-6 Sol | 73.1% | OpenAI |
| 25th NEW | Muse Ember | 72.7% | Meta |
| 26th NEW | GPT-6.1 Terra | 72.2% | OpenAI |
| 27th | GPT-5.6 Sol Pro (xhigh) | 71.7% | OpenAI |
| 28th | Qwen 3.7 Max | 70.4% | Alibaba |
| 29th | Grok 4.5 | 70.0% | xAI |
| 30th NEW | GPT-6.2 Sol | 69.9% | OpenAI |
| 31st | GPT-5.5 | 69.0% | OpenAI |
| 32nd | Claude Opus 4.6 | 67.6% | Anthropic |
| 33rd | DeepSeek V4.1 Flash | 66.7% | DeepSeek |
| 34th | GLM 5.3 | 66.2% | Z.ai |
| 35th | Claude Opus 4.8 | 64.8% | Anthropic |
| 36th | GPT-5.6 Sol (xhigh) | 64.8% | OpenAI |
| 37th | Qwen 3.6 Max Preview | 63.0% | Alibaba |
| 38th | Qwen 3.8 2.4T A95B | 62.5% | Alibaba |
| 39th | Gemini 2.5 Pro (06-05) | 62.4% | |
| 40th | Claude Opus 4.5 | 62.0% | Anthropic |
| 41st | Claude Opus 4.7 | 61.7% | Anthropic |
| 42nd | GPT-5 Pro | 61.6% | OpenAI |
| 43rd | Gemini 3 Flash Preview | 61.1% | |
| 44th | DeepSeek V4 Flash | 61.1% | DeepSeek |
| 45th | Kimi K3 (max) | 60.7% | Moonshot AI |
| 46th | Claude Sonnet 5 | 60.6% | Anthropic |
| 47th | Grok 4 | 60.5% | xAI |
| 48th | Qwen 3.8 27B | 60.2% | Alibaba |
| 49th | Claude 4.1 Opus | 60.0% | Anthropic |
| 50th NEW | Claude Haiku 5.5 | 59.2% | Anthropic |
| 51st | Claude 4 Opus | 58.8% | Anthropic |
| 52nd | GLM 5.2 | 58.8% | Z.ai |
| 53rd | Kimi K2.7 Code | 57.9% | Moonshot AI |
| 54th | GPT-5.2 Pro (xhigh) | 57.4% | OpenAI |
| 55th | GPT-5 (high) | 56.7% | OpenAI |
| 56th | Grok 4.1 Fast | 56.0% | xAI |
| 57th | GLM 5.1 | 55.1% | Z.ai |
| 58th | Ox Alpha | 55.1% | Unknown |
| 59th | Claude 4.5 Sonnet | 54.3% | Anthropic |
| 60th | GPT-5.1 (high) | 53.2% | OpenAI |
| 61st | GLM 5 | 53.2% | Zhipu AI |
| 62nd | o3 (high) | 53.1% | OpenAI |
| 63rd | DeepSeek 3.2 Speciale | 52.6% | DeepSeek |
| 64th | Gemini 2.5 Pro (03-25) | 51.6% | |
| 65th | DeepSeek V4 Pro | 50.9% | DeepSeek |
| 66th | Inkling | 50.0% | Thinking Machines |
| 67th | GPT-5.6 Terra (xhigh) | 48.9% | OpenAI |
| 68th | GLM 4.7 | 47.7% | Z.ai |
| 69th | Kimi K2.5 | 46.8% | Moonshot AI |
| 70th | GPT-5.6 Luna (xhigh) | 46.8% | OpenAI |
| 71st | Claude 3.7 Sonnet (thinking) | 46.4% | Anthropic |
| 72nd | DeepSeek V4 Flash (preview) | 46.3% | DeepSeek |
| 73rd | GPT-5.2 (high) | 45.8% | OpenAI |
| 74th | MiniMax M3 | 45.8% | MiniMax |
| 75th | Claude 4 Sonnet (thinking) | 45.5% | Anthropic |
| 76th | Claude 3.7 Sonnet | 44.9% | Anthropic |
| 77th | o1-preview | 41.7% | OpenAI |
| 78th | Nemotron 3 Ultra | 41.7% | Nvidia |
| 79th | Claude 3.5 Sonnet 10-22 | 41.4% | Anthropic |
| 80th | Gemini 2.5 Flash (latest) | 41.2% | |
| 81st | DeepSeek R1 05/28 | 40.8% | DeepSeek |
| 82nd | o1-2024-12-17 (high) | 40.1% | OpenAI |
| 83rd | DeepSeek V3.1 | 40.0% | DeepSeek |
| 84th | Kimi K2 Thinking | 39.6% | Moonshot AI |
| 85th | o4-mini (high) | 38.7% | OpenAI |
| 86th | o1-2024-12-17 (med) | 36.7% | OpenAI |
| 87th | Grok 3 | 36.1% | xAI |
| 88th | Qwen 3.6 Flash | 35.2% | Alibaba |
| 89th | MiniMax M2.1 | 34.7% | MiniMax |
| 90th | GPT-4.5 | 34.5% | OpenAI |
| 91st | Gemini-exp-1206 | 31.1% | |
| 92nd | Qwen3 235B-A22B | 31.0% | Alibaba |
| 93rd | DeepSeek R1 | 30.9% | DeepSeek |
| 94th | Gemini 2.0 Flash Thinking | 30.7% | |
| 95th | Llama 4 Maverick | 27.7% | Meta |
| 96th | Claude 3.5 Sonnet 06-20 | 27.5% | Anthropic |
| 97th | DeepSeek V3 03-24 | 27.2% | DeepSeek |
| 98th | Gemini 1.5 Pro 002 | 27.1% | |
| 99th | GPT-4.1 | 27.0% | OpenAI |
| 100th | Kimi K2 | 26.3% | Moonshot AI |
| 101st | GPT-4 Turbo | 25.1% | OpenAI |
| 102nd | MiniMax M2 | 25.0% | MiniMax |
| 103rd | Claude 3 Opus | 23.5% | Anthropic |
| 104th | Llama 3.1 405b instruct | 23.0% | Meta |
| 105th | o3-mini (high) | 22.8% | OpenAI |
| 106th | Grok 2 | 22.7% | xAI |
| 107th | Mistral Large v2 | 22.5% | Mistral |
| 108th | GPT-OSS 120B | 22.1% | OpenAI |
| 109th | Mistral Large 3 | 20.4% | Mistral |
| 110th | Llama 3.3 70b instruct | 19.9% | Meta |
| 111th | DeepSeek V3 | 18.9% | DeepSeek |
| 112th | Gemini 2.0 Flash Exp | 18.9% | |
| 113th | o1-mini | 18.1% | OpenAI |
| 114th | GPT-4o 08-06 | 17.8% | OpenAI |
| 115th | Command R+ | 17.4% | Cohere |
| 116th | GPT-4o mini | 10.7% | OpenAI |
| 117th | GPT-3.5 Turbo | 8.0% | OpenAI |
benchmark settings
temperature: 0.7, top-p: 0.95 (except o1 series)
*See Human Evaluation section of Report for details on how we calculated Human Baseline.
**We try an engineered prompt to optimize benchmark specific performance. See LLM Eval section of Report for details.
Video Summary
Go to the Patreon for the latest, fully-explained update
Please reach out to [aiexplained@outlook.com](mailto:aiexplained@outlook.com) for inquiries, business questions or feedback about SimpleBench. This page was was adopted from the Nerfies project page.