method · first seen 3 Nov 2026
bits per byte
Also: BPB
Tokenizer-independent language model evaluation metric measuring compressed code length in bits per byte of text.
Mentions per week
PapersCodeDiscussion
Earliest sources
- Tokenizer Myths and Where To Find ThemPapers · 3 Nov 2026 · Rylan Schaeffer, Sanmi Koyejo
- Adapting Small Language Models to Romanized Nepali: Tokenizer Fertility, Reference Likelihood, and Generation MetricsPapers · 9 Nov 2026 · Pujan Pant, Sagar Newpane, Hari Sitaula, Manish Sakhakarmy
Latest sources
- Adapting Small Language Models to Romanized Nepali: Tokenizer Fertility, Reference Likelihood, and Generation MetricsPapers · 9 Nov 2026 · Pujan Pant, Sagar Newpane, Hari Sitaula, Manish Sakhakarmy
- Tokenizer Myths and Where To Find ThemPapers · 3 Nov 2026 · Rylan Schaeffer, Sanmi Koyejo