跳到正文
𝕏 Rohan Paul(x.com)·· 9 天前品質評分57
重點導讀

MoE 模型常把不同工作交給不同專家,但專家的能力可能分散在專家內的神經元裡。

  • Tencent 的研究用 RAMP 衡量神經元對不同領域的貢獻,再依分數剪掉較不重要的部分。
  • 測試涵蓋 6 個模型,留下 75% 的神經元時,表現仍保有原本的 91% 到 98%。留下 50% 時,GPU 記憶體用量減少 36% 到 43%,批次預填速度提高 17% 到 39%。結果顯示,壓縮 MoE 模型時,逐個判斷神經元的貢獻可能比整個移除專家更能保留原有表現。

AI 伴隨問答

自備 Key 深度分析

針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:

Rohan Paul
Rohan Paul@rohanpaul_ai
·9 天前
𝕏

New Tencent paper shows domain skills (like math or coding ) in Mixture of Experts models sit in scattered neurons across expert.

so cutting weak neurons instead of whole experts shrinks the models while keeping most skills.

In their experiment, Mixture of Experts AI models kept 91% to 98% of their performance after their least useful 25% of expert neurons were cut.

MoE models send each token to a few experts, which suggests each expert handles a domain like math or code. Other work finds routing mostly tracks token type instead.

They scored each neuron's routed contribution with a measure called RAMP. Neurons picked for one domain overlapped 58% to 74% across samples of that domain, versus 15% to 24% across domains, and sat in mixed experts.

RAMPr prunes by these scores across all experts and kept 91% to 98% of original performance with 75% of neurons left, across 6 models. With 50% left, GPU memory fell 36% to 43% and batched prefill throughput rose 17% to 39%.

If you compress MoE models, prune neurons inside experts by their routed contribution instead of treating each expert as a single unit.

推文來源 · X (Twitter)在 X 參與討論

讀者討論

理性交流 · 即時討論