跳到正文
𝕏 Rohan Paul(x.com)·· 5 天前品質評分51
重點導讀

用 AI 助理幫忙挑選下一個模型設定,聽起來能省下試錯時間,但錯誤建議也可能把搜尋帶偏。

  • Amazon 與麻薩諸塞大學的論文比較加入 LLM 評審和不加入時的結果:Llama-3.1-8B 在 8 個資料集中的 7 個表現較差,AUC 最多下降 0.245。
  • 理想的評審最多隻提高約 0.005 AUC。作者也發現,使用 250 組設定、進行 20 輪搜尋時,傷害大多會減少。團隊可用相同的搜尋預算做對照,確認評審有幫助再採用。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

Rohan Paul
Rohan Paul@rohanpaul_ai
·5 天前
𝕏

New Amazon + University of Massachusetts paper finds that adding an LLM critic to a search loop has small upside, about 0.005 AUC at best, and much larger downside.

Weak LLM critic can push AutoML search below plain random search

Agent systems often add an LLM critic that reads past results and tells the search what to try next. Teams check whether it helps, rarely whether it hurts.

They ran the same tabular model search with and without a critic, over a fixed pool of 96 pipeline setups. Random critic mistakes cost almost nothing since the loop keeps its best result, but advice that steadily points the wrong way wastes tries.

A Llama-3.1-8B critic scored below the no-critic baseline on 7 of 8 datasets, losing up to 0.245 AUC. An ideal critic gained at most about 0.005 AUC, and the harm mostly faded with 250 setups and 20 rounds.

Before trusting an LLM critic, even a large one, run the same budget without it and drop it if it loses.

推文來源 · X (Twitter)在 X 參與討論

讀者討論

理性交流 · 即時討論