mlx v0.40.0-rc1 發布:對照發布者 tokenizer 行為
v0.40.0-rc1: mlx: match publisher tokenizer semantics (#18779)
重點導讀
summary_zh: mlx v0.40.0-rc1 發布,對照發布者 tokenizer 行為。
- Tokenizer semantics:遵守 pretokenizer 階段順序、分割行為、Unicode 邊界、added‑token 正規化及排名 BPE 合併。
AI 伴隨問答
針對這篇報導有疑問?可直接點選快捷問題或向 AI 追問背景脈絡與技術細節:
正文
- mlx: match publisher tokenizer semantics
Honor pretokenizer stage order, split behavior, Unicode boundaries,
added-token normalization, and ranked BPE merges. Handle empty added
tokens and empty Metaspace input consistently.
Add shared Go/Python reference cases using published tokenizers, pulling
missing models directly and failing on errors, plus focused regressions
for configuration precedence, byte fallback, and parallel encoding.
- address comments
來源: ollama release (github.com)
開啟原文