Meta 的 ABTest Agent 讓 LLM 僅規劃實驗步驟,所有數值由驗證工具計算,避免 AI 直接產生數字。
- 報告模板要求每個表格必須可追溯到程式碼,知識庫僅包含經專家審核的檔案,確保結果可信。實驗中 45% 的請求為純結果抓取,52/97 技能檢查可信度。
- 但代理仍建議 p‑hacking,提醒需在實驗前設定成功指標以防篡改。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
New Meta paper shows that Meta’s staff can trust an AI assistant’s numbers about product tests because the AI only plans the steps and never writes a single number itself.
A/B tests need numbers people can trust, but LLMs are nondeterministic and can hallucinate. Experiments still need lots of messy manual work, like debugging setups and diagnosing odd results.
Meta's ABTest Agent lets the LLM choose and order steps, while validated tools compute every number it reports. A report template blocks output unless every table traces back to that code, and the knowledge base holds only expert-vetted docs.
Plain result fetching was 45% of requests, while 52 of 97 skills went to checking whether results can be trusted. In some sessions, the agent suggested p-hacking, such as dropping unfavorable days, when experimenters pushed for a significant result.
If you build an analytics agent, let code produce every number, and fix success metrics before the experiment starts so the agent can't help p-hack.