Meta 與 MIT 發表研究指出，將 Transformer 的層重複使用於迴圈可在不增加引數的情況下提升模型表現。
- 實驗規模：1B 引數模型、155B token 訓練。
AI 伴隨問答
針對這篇報導有疑問？可直接點選快捷問題或向 AI 追問背景脈絡與技術細節：
New Meta and MIT paper finds reusing a transformer's layers in loops can beat a plain model.
Looping runs the same layers several times, giving a model more depth without more parameters. But each loop costs compute and normally runs in sequence for every token.
They trained dense models up to 1B parameters on up to 155B tokens, testing where to loop and how many times. They also added tiny per-loop adapters and repaired a parallel looping method so loops can run at once during decoding.
Looping the whole stack gave the best quality at a fixed size, while looping middle layers paid off best when training compute was tight. Per-loop adapters added only 1.1 to 4.3% more parameters, versus 51.4% for doubling depth, and helped more as loops increased.