AI 模型在長對話中會用掉大量記憶體,草稿模型也需要推論系統配合才能投入實際使用。
- vLLM 團隊公佈為 Kimi K3.1 Pro 設計的 DSpark,這項工作由 Inferact 與 Red Hat Speculators 團隊共同開發。每個草稿層都使用 2K 的滑動視窗,讓草稿 KV 記憶體用量維持在固定範圍,即使上下文長達 1M tokens 也一樣。HumanEval 測試中,DSpark 的接受長度比預覽版提高最多 30%。
- vLLM 重新設計的 KV 管理也支援分離式推論與不同的平行配置。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
More accepted tokens. Bounded draft KV memory. 🔥
The new DSpark for @Kimi_Moonshot’s Kimi K3.1 Pro, jointly developed by @Inferact and the @RedHat_AI Speculators team with vLLM + Speculators, brings:
• Up to 30% higher acceptance length vs. the preview (HumanEval).
• A 2K sliding window in every draft layer, keeping draft KV memory bounded even at 1M-token context.
Efficient draft architectures need an inference stack that can serve them. vLLM’s redesigned KV management supports P/D disaggregation and different parallelism configurations, bringing sliding-window draft models into production.
Comparison: DSpark k=7, temperature 1.0, top_p 0.95. Acceptance length includes the bonus token.
Weights + vLLM config below. 👇