summary_zh: 卡內基梅隆大學研究團隊提出,預訓、微調與 RL 可統一為單一訓練規則,並證實混合資料優於分階段訓練。
- 訓練方法:將 RL 形式 MaxRL 改為模型自身正確答案的似然訓練,允許在任何比例混合預訓文本、精選資料與 RL 範例。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
The separate stages of AI training can collapse into a single method.
New Carnegie Mellon paper shows that pretraining, fine-tuning, and RL can share a single training rule, and blending their data often beats separate stages.
LLMs are usually trained in stages: pretraining, midtraining, fine-tuning, then RL, each with its own algorithm. Hard switches between stages can also erase earlier skills.
The authors recast a form of RL called MaxRL as likelihood training on the model's own correct answers. That lets one trainer mix pretraining text, curated data, and RL samples in any proportion over time.
On reasoning puzzles with a 124M model, the best blended runs beat the best staged run and kept general skills better. After later coding RL, a 0.76B model that blended in 5% midtraining data during math RL reached 32.69% on MBPP+, versus 29.81% for staged training.
Treat your training pipeline as a data-mixing schedule, and try blending earlier data into RL instead of switching stages abruptly.