跳到正文
𝕏 Rohan Paul(x.com)·· 15 天前品質評分38
重點導讀

Salesforce 研究發現,從大型外部教師蒸餾成本高於單純 RL,且將 RL 訓練的域專家蒸餾至一模型可節省 42% 計算量。

  • - GRPO:在外部教師情境下普遍勝過蒸餾。
  • - 蒸餾:每個訓練 token 需教師前向傳播,往往導致回應長度膨脹。
  • - 建議:先用 RL 訓練各域專家,再蒸餾成單一模型,而非直接從大型教師蒸餾。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

Rohan Paul
Rohan Paul@rohanpaul_ai
·15 天前
𝕏

New Salesforce paper finds that distilling from a big external teacher usually costs more compute than plain RL, while merging RL-trained domain experts saves 42%.

On-policy distillation lets a student learn from a teacher's per-token feedback on its own outputs, and many expect this dense signal to beat RL. But every training token also needs a teacher forward pass.

They compared distillation and GRPO by held-out accuracy per training FLOP, varying task difficulty, model sizes, domains, and training settings.

With an external post-trained teacher, GRPO generally won, and distillation often inflated response length. Distilling per-domain GRPO experts into 1 student matched joint GRPO on math plus code with 42.0% fewer total FLOPs.

To save compute, train per-domain experts with RL and distill them into 1 model, rather than distilling from a big external teacher.

推文來源 · X (Twitter)在 X 參與討論

讀者討論

理性交流 · 即時討論