斯坦福研究指出，單一分詞器評估指標無法在所有情況下保持 tokenizer‑agnostic，研究者在比較模型時應結合多指標與實際任務表現。
- - 核心發現：即使未訓練的模型，BPB 也因分詞器不同而差異。
- 在已訓練模型中，BPB 與任務準確率的相關性僅為 +0.02 至 −0.30。
AI 伴隨問答
針對這篇報導有疑問？可直接點選快捷問題或向 AI 追問背景脈絡與技術細節：
New Stanford paper finds that no metric can be tokenizer-agnostic in every sense, so anyone comparing tokenizers should not trust a single score and should also check whether the model answers correctly.
To compare models with different tokenizers, researchers usually report bits-per-byte, or BPB, which divides the loss by bytes instead of tokens. Many papers call it tokenizer-agnostic.
They scored untrained models that guess uniformly, plus released models that share one recipe but use different tokenizers. They also rechecked a published claim that another supposedly tokenizer-neutral metric agrees with BPB.
Models that learned nothing still got different BPB by tokenizer, with English text scoring best near a 24K vocabulary, while accuracy stayed at chance. On models trained with different tokenizers, correlations between BPB and task scores ran only from +0.02 to −0.30.
When comparing tokenizers, don't treat lower BPB as proof of a better model, and check task accuracy on the same text before picking a vocabulary.