跳到正文
Hacker News (official RSS)(arxiv.org)·· 19 天前品質評分41

多令牌預測(Multi-Token Prediction)正確完成

Multi-Token Prediction Done Right

重點導讀

summary_zh: 研究人員提出 Soft-MTP,一行程式碼即可修正多令牌預測(MTP)梯度中的雜訊,提升模型從 167M 到 1.2B 引數的下游效能與推理速度。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

正文

View PDF HTML (experimental)

Abstract:Multi-Token Prediction (MTP) is now a common ingredient in language-model pre-training, as it improves downstream performance and enables faster inference through speculative decoding. We identify a flaw in standard MTP training, in which a residual component of the MTP gradient interferes with both next-token and multi-token learning, increasingly so with scale. Specifically, the MTP cross-entropy gradient can be decomposed exactly into a KL self-distillation term that pulls each MTP head toward the model's own next-token prediction (NTP) distribution and improves speculative decoding, and a residual that has a low signal-to-noise ratio and buries the useful KL signal. We introduce the KL gradient-recovery trick: when two distributions share a linear output head, the gradient of their KL divergence can be recovered exactly by combining their cross-entropy gradients through straight-through gradient routing. Unlike minimizing the KL directly, this trick requires no vocabulary-sized logit matrices and is compatible with standard fused cross-entropy kernels. This enables Soft Multi-Token Prediction (Soft-MTP), a one-line code change to MTP that removes the noisy residual and backpropagates only the KL self-distillation signal by reusing the existing NTP and MTP cross-entropy computations. Soft-MTP supports both DeepSeek- and Meta-style MTP, adds no parameters, and incurs no measurable overhead. From 167M to 1.2B parameters, Soft-MTP improves over standard MTP with lower NTP perplexity, faster speculative decoding and higher zero-shot accuracy at every scale, lower MTP perplexity from 689M onward, and stronger math and code reasoning after fine-tuning at 1.2B. Soft-MTP also matches, if not exceeds, direct KL divergence minimization on most metrics. Soft-MTP is, in effect, a free lunch for MTP optimization. Our code can be found at this https URL.
Comments: Preprint
Subjects: Machine Learning (cs.LG)
Cite as: arXiv:2610.38331 [cs.LG]
  (or arXiv:2610.38331v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2610.38331

arXiv-issued DOI via DataCite

Submission history

From: Jayden Teoh [view email]
[v1] Wed, 28 Oct 2026 06:31:05 UTC (927 KB)

讀者討論

理性交流 · 即時討論