CMU 等團隊提出一種統一訓練框架，將預訓練、微調和 RL 統合於單一最大似然演算法。
- \n- 核心發現：統一訓練在多個任務上優於分階段訓練。
- \n- 實驗範例：124M 模型在推理謎題上表現更佳，0.76B 模型在數學 RL 任務中保持準確度。
- \n- 實際影響：可簡化模型訓練流程，減少多階段設計與資源開銷。
Highly recommended paper from CMU and others on how foundation models get trained.
It fundamentally asks whether pretraining, midtraining, SFT, and RL really need to be separate stages.
They show that this separation isn't fundamental.
All of these stages can run under a single likelihood-based algorithm, which they call unified training.
The key step treats suitable RL objectives as maximum-likelihood fitting to model-generated targets. RL generations and static data then share one loss and one trainer.
This changes the design problem. You now choose which data to train on, in what proportions and when. Staged training becomes one special case.
They first trained a 124M model from scratch on reasoning puzzles. The best unified runs beat staged training on the RL tasks. They also beat it on larger puzzles the model never saw.
Those runs reused the midtraining corpus up to 4 times. Staged training lost accuracy when midtraining ran past its best budget.
Under a fixed compute budget, unified training had the best accuracy at the smallest and largest budgets. Staged training led at intermediate budgets.
The real-world test used a 0.76B model on math RL, with 0% to 5% midtraining data mixed in. Math accuracy stayed similar while retention improved. Coding accuracy dropped as the mix grew.
Continual learning gave the clearest result. After math training, all unified models beat staged training on final coding accuracy, despite starting lower. The 5% mix did best.
The experiments are small-scale. The authors have not found a mixture or schedule that works best across the board.
Chat with Paper: https://t.co/CPpctbQJiS