資料裡明明沒有真實效果,AI 助理反覆嘗試不同分析後,還是可能挑出看起來顯著的結果。
- 密西根大學研究者讓 4 個前沿 AI 助理分析模擬資料,發現它們有 54% 到 98% 的測試報出低於 0.05 的 p 值,但這些都是假發現。研究也用子樣本重跑分析,估算 AI 助理挑選分析方式造成的偏差。
- 校正後,模擬資料的假陽性降到 5.1%,另外在 3 篇已發表論文的資料上降到 5.9%。因此,若分析方式沒有事先固定,單看 AI 助理回報的 p 值可能會誤信假結果。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
研究呈現 AI 助理反覆嘗試分析後挑出顯著結果的偏差,也比較了用子樣本校正前後的假陽性比例。
New University of Michigan paper finds that AI agents asked to analyze data where nothing was going on still reported a discovery in up to 98% of cases.
That anyone using coding agents for data analysis should not trust their p-values as reported
And replaying them on subsamples fixes it.
Researchers now hand data analysis to coding agents that try many models before reporting 1. Classic p-values assume the test was chosen before seeing the data.
They asked 4 frontier agents for 1 compelling finding in simulated data with no real effects. To correct the bias, they rerun the agent on subsamples and measure how much more often it picks its analysis when the result looks good.
The agents reported a p-value under 0.05 in 54% to 98% of runs, all false. The correction cut false positives to 5.1% in simulations and 5.9% on data from 3 published papers.
If agents run your data analysis, distrust their p-values unless the analysis was fixed in advance or corrected for the agent's choices.