ollama release· dhiltgen·· 3 小時前AI 評分23
mlx v0.40.0-rc1 發布：對照發布者 tokenizer 行為
v0.40.0-rc1: mlx: match publisher tokenizer semantics (#18779)
AI 導讀
summary_zh: mlx v0.40.0-rc1 發布，對照發布者 tokenizer 行為。
- Tokenizer semantics：遵守 pretokenizer 階段順序、分割行為、Unicode 邊界、added‑token 正規化及排名 BPE 合併。
正文
- mlx: match publisher tokenizer semantics
Honor pretokenizer stage order, split behavior, Unicode boundaries,
added-token normalization, and ranked BPE merges. Handle empty added
tokens and empty Metaspace input consistently.
Add shared Go/Python reference cases using published tokenizers, pulling
missing models directly and failing on errors, plus focused regressions
for configuration precedence, byte fallback, and parallel encoding.
- address comments
來源：ollama release · github.com