Cohere 新論文顯示即使批次為 1，精細 MoE 模型的 Speculative Decoding 仍有效。
- 效能提升：相鄰 token 共享 20%–45% 的專家，遠高於預期的 4%–12%，使檢查額外 token 成本降低。
AI 伴隨問答
針對這篇報導有疑問？可直接點選快捷問題或向 AI 追問背景脈絡與技術細節：
此研究證實在低批次 MoE 服務中，speculative decoding 能大幅提升效能，對於需要即時回應的代理工作負載具有實務價值。
New @cohere paper finds that speculative decoding works well on fine-grained MoE models even at batch size 1, where prior work predicted small gains.
Finds that in mixture-of-experts models, neighboring tokens share 20% to 45% of their experts, against 4% to 12% expected, which makes speculative decoding cheaper than predicted.
Speculative decoding lets a small drafter propose tokens that the big model checks in 1 pass. For MoE models, prior work said checking several tokens loads many extra experts, limiting gains at low batch sizes.
They tested 10 MoE models from 30B to 1T parameters and profiled them on H100 GPUs. Consecutive tokens reuse many experts, and small experts use GPU bandwidth well, so checking extra tokens costs little.
Adjacent tokens shared 20% to 45% of their experts, against 4% to 12% expected from independent routing. For Qwen3-235B in FP8, the verification pass ran at 89% of a normal decoding step's speed, against 52% predicted.
For low-batch MoE serving, such as agent workloads, benchmark speculative decoding on your hardware instead of ruling it out.