重點導讀
開源社群在 TPU v7 上為 Muse Spark Open 建立 megakernel,於 AgentX(1M context)達到每位使用者 433 tok/s、每顆晶片最高 25.9K tok/s。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
PyTorch@PyTorch
·12 天前
Great to see the open source community pushing inference performance forward on PyTorch:
A megakernel for Muse Spark Open on TPU v7 got 433 tok/s per user and up to 25.9K tok/s per chip on AgentX (1M context). All 62 layers of the 816B MoE run as one Pallas program on a single TPU v7 host, with SGLang on the TorchTPU PyTorch backend.
- 1.94x faster per token than our previous path
- DFlash2 speculative decoding: 2.0x per token
- fp8 KV, HiCache and session-aware eviction for long agent sessions
Read the full technical deep dive here: https://t.co/xmxOjlrZtK
推文來源 · X (Twitter)在 X 參與討論