做機器學習研究時,研究者得想辦法判斷實驗有沒有進展;FrontierML 想測試 AI 助理能不能自己完成這件事。
- 每個任務會藏起評分目標,讓 AI 助理在 Lab0 使用 H100 GPU 工作 12 小時,自己設計進度指標、做實驗並判斷這些指標有多可信,最後再提交模型或方法評分。
- Synthetic Sciences 表示,受測的 5 個前沿 AI 模型在 25 個研究問題中,都沒有達到強成果門檻。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
Synthetic Sciences, a Y Combinator startup, has launched FrontierML, a benchmark so hard that none of the 5 frontier AI models it tested hit the strong-result bar on any of its 25 ML research problems.
They hide each objective, so agents must build their own tests and decide how much to trust them.
This is a test of whether AI agents can do machine learning research on their own without ever seeing the score they’ll be graded on.
Each agent gets 12 hours and real GPUs, and has to work out for itself how to measure progress.
FrontierML runs each task as a loop with the objective kept out of reach:
> The agent gets an open-ended ML problem and 12 hours in Lab0
> It designs a local proxy for progress
> It runs experiments on H100s and calibrates how far that proxy can be trusted
> It submits a model or method
> FrontierML scores it on a held-out evaluation
Synthetic Sciences argues step 3 is where much of empirical research happens. Here, a lot of the score rides on it.