phenomenon · first seen 17 Oct 2026
tokenizer fertility
The average number of subword tokens produced per word, indicating tokenization efficiency across languages.
Mentions per week
PapersCodeDiscussion
Earliest sources
- Efficiency Measured in English: Tokeniser Coverage Bounds Small Language Models on Indic ScriptsPapers · 17 Oct 2026 · Anna Pramod
- DeepSEA: A Multilingual Evaluation Benchmark for Southeast AsiaPapers · 19 Oct 2026 · Jiuzhou Han, Abhishek Bijay Mishra, Trevor Cohn, Shubham Mittal
- Adapting Small Language Models to Romanized Nepali: Tokenizer Fertility, Reference Likelihood, and Generation MetricsPapers · 9 Nov 2026 · Pujan Pant, Sagar Newpane, Hari Sitaula, Manish Sakhakarmy
- A Corpus-Driven SentencePiece Unigram Tokenizer for Sindhi: Design and Benchmark Against Fourteen Existing TokenizersPapers · 15 Nov 2026 · Ali Nawaz, Muqadas Fareed, Soonh Sadaf
Latest sources
- A Corpus-Driven SentencePiece Unigram Tokenizer for Sindhi: Design and Benchmark Against Fourteen Existing TokenizersPapers · 15 Nov 2026 · Ali Nawaz, Muqadas Fareed, Soonh Sadaf
- Adapting Small Language Models to Romanized Nepali: Tokenizer Fertility, Reference Likelihood, and Generation MetricsPapers · 9 Nov 2026 · Pujan Pant, Sagar Newpane, Hari Sitaula, Manish Sakhakarmy
- DeepSEA: A Multilingual Evaluation Benchmark for Southeast AsiaPapers · 19 Oct 2026 · Jiuzhou Han, Abhishek Bijay Mishra, Trevor Cohn, Shubham Mittal
- Efficiency Measured in English: Tokeniser Coverage Bounds Small Language Models on Indic ScriptsPapers · 17 Oct 2026 · Anna Pramod