跳到正文
Hacker News (official RSS)(arxiv.org)·· 8 天前品質評分68

研究提出降低 MoE 長上下文訓練記憶體尖峰的方法

Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training

重點導讀

MoE 模型在長上下文或大批次訓練時,只要有一個步驟用掉太多 GPU 記憶體,整個訓練就會停下來。

  • 這篇研究分別調整專家分流、詞彙投影、檢查點資料和最佳化器更新時的資料處理方式,讓 GPU 一次需要存放的資料量保持固定,並稱不會改變模型的損失與梯度。組合使用後,研究者讓引數量為 120B 到 667B 的 MoE 模型以 1M 上下文長度訓練。
  • 相較調整過的 FSDP2 基準,涵蓋範圍達 8–32 倍,速度最高達 10.4 倍。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

正文

View PDF HTML (experimental)

Abstract:Training a Mixture-of-Experts (MoE) model at long context or large batch size fails as soon as any one component's peak allocation exceeds device memory, so the target is every peak at once, not the average footprint. Four are left unbounded by the parallelism plans in common use, and each grows differently: expert dispatch with the routing matrix, the vocabulary projection with tokens times vocabulary, gradient checkpoint boundaries with depth times sequence length, and optimizer state with parameter count. Which one runs out first changes with the model, the context length, and the device count, so lowering the largest only exposes the next. We bound all four with schedules whose GPU working set is fixed at launch: PipelinedLLEP extends least-loaded expert parallelism with a cap on the tokens each source contributes to a dispatch chunk, Ring-DTP circulates activations or weight shards around a ring at the vocabulary projection and folds each block of logits into an online log-sum-exp, Selective checkpoint offload (SCO) keeps the one long-lived tensor of each checkpoint boundary in CPU memory, and OffloadStreamAdamW turns the serial CPU Adam update of optimizer offload into a bucket pipeline. All four change only the order and granularity of computation and data movement, so the loss and gradients stay exact. In matched component tests, they cut the MoE dispatch peak by up to $59.3\%$ without losing throughput, the vocabulary projection peak by $86.6\%$, and the offloaded optimizer step by $2.05\times$ faster. Composed on MoE models from 120B to 667B parameters, they train at 1M context length, $8$--$32\times$ the reach of a tuned FSDP2 baseline, and up to $10.4\times$ its throughput.
Comments: Preprint
Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
Cite as: arXiv:2609.14306 [cs.DC]
  (or arXiv:2609.14306v1 [cs.DC] for this version)
  https://doi.org/10.48550/arXiv.2609.14306

arXiv-issued DOI via DataCite

Submission history

From: Xuan Phi Nguyen [view email]
[v1] Sun, 13 Sep 2026 06:00:46 UTC (1,012 KB)

讀者討論

理性交流 · 即時討論