method · first seen 17 Oct 2026
byte-fallback tokenization
Tokenizer strategy that encodes characters unseen in the vocabulary as raw UTF-8 bytes.
Mentions per week
PapersCodeDiscussion
Earliest sources
- Efficiency Measured in English: Tokeniser Coverage Bounds Small Language Models on Indic ScriptsPapers · 17 Oct 2026 · Anna Pramod
Latest sources
- Efficiency Measured in English: Tokeniser Coverage Bounds Small Language Models on Indic ScriptsPapers · 17 Oct 2026 · Anna Pramod