method · first seen 4 Oct 2026
byte pair encoding
Also: BPE, byte-level BPE, byte-level byte pair encoding
A subword tokenization algorithm that iteratively merges the most frequent adjacent symbol pairs.
Byte pair encoding is the subword tokenization method behind most current LLM vocabularies: it starts from characters or bytes and repeatedly merges the most frequent adjacent pair into a new token. The sources do not introduce BPE. They use it as a baseline or build on it. Papers cover compositional multi-word tokens, per-language tokenizer mixtures, pre-tokenizer fixes for Devanagari and other scripts, vocabulary trimming for on-device models, and formal results on lookbehind and NP-hardness of merge selection. A few repos wrap or reuse it, including a SIMD tokenizer library and a copy of BPEmb embeddings.
The rise reflects ongoing tokenizer research, especially on multilingual token cost, rather than new adoption of BPE itself. The more specific variants in these papers, such as compositional or multi-word tokenization, are the parts worth tracking separately.
Analyst note (established), 17 Nov 2026.
Mentions per week
PapersCodeDiscussion
Rank 71 on 14 Nov 2026 · 14 mentions in 28 days · 13 independent groups · papers → code · first seen 6 weeks ago · acceleration +1.82
Earliest sources
- More Than Words: Compositional Tokenization for Efficient Language ModelsPapers · 4 Oct 2026 · Yuval Reif, Guy Kaplan, Roy Schwartz
- actual-computer/toksCode · 5 Oct 2026 · actual-computer · 118 stars
- Phonologically Informed Tokenization for German Speech Recognition: A Cross-Domain StudyPapers · 8 Oct 2026 · Christopher Witzl, Tobias Bocklet, Korbinian Riedhammer
- Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued PretrainingPapers · 8 Oct 2026 · Aarjan Chaudhary
- AdaptTok-Indic: Script-Aware Adaptive Tokenization for Indic LanguagesPapers · 8 Oct 2026 · Binayak Mukherjeea, Moitreya Nandi, Arindam Panja
- Latent Core Tokenizer: Compress, but MeaningfullyPapers · 8 Oct 2026 · Felermino D. M. A. Ali, Millicent Ochieng, Ogbemi Ekwejunor-Etchie, Ade Famoti,…
Latest sources
- The Vocabulary Tax: Safely Trimming Unused Tokens from LLMs for On-Device UsePapers · 15 Nov 2026 · Sattaya Singkul
- ActionCodec2: A Streaming Action TokenizerPapers · 11 Nov 2026 · Zibin Dong, Yicheng Liu, Shiduo Zhang, Tianyuan Yuan, Baijun Ye, Jinqiu Wang, Ha…
- What Do Tokenizer Comparisons Measure? Disentangling Granularity, Root Assignment, and Word-Level Readout across Amharic and HausaPapers · 10 Nov 2026 · Abdul Rehman Antall, Baidya Nath Saha
- Pair-Merge Tokenisation Is Hard to Approximate on Binary Strings and Strongly NP-Hard on Unary DatasetsPapers · 10 Nov 2026 · Kundan Baliga, Aiden Xie, Aditya Byju
- The Left Ripple of Byte-Pair Encoding: Certified Lookbehind for Every DictionaryPapers · 9 Nov 2026 · Nicklas Nidhögg
- A Multi-Word Tokenizer for 204 Languages That Needs Fewer Tokens Than GPT-4o's, Including on EnglishPapers · 9 Nov 2026 · Johin Johny Arimbur, Ankur Ashit Ghosh, Shervin Sam
- MoTo: Mixture of Tokenizers Towards Fair Multilingual Language ModelingPapers · 8 Nov 2026 · Gül Sena Altıntaş, Colin Raffel
- Compact Greek Language Models from Scratch: Pretraining 500M- and 1B-Parameter Decoder-Only TransformersPapers · 6 Nov 2026 · Konstantinos Perifanos, Anne Laure Poite, Giannis Tsilikas, Ilias Katis, Stavros…
- boepeanjafay/bpembCode · 5 Nov 2026 · boepeanjafay · 50 stars
- Greedy Byte-Pair Encoding Achieves 0.391 of the Optimum, Past the One-Third Barrier for Packing-Based AnalysesPapers · 5 Nov 2026 · Kundan Baliga, Aiden Xie, Aditya Byju
- AssameseLM-E2B: An Open-Weight Assamese Language Model for Efficient Consumer-Grade DeploymentPapers · 4 Nov 2026 · Pragyan Phukan
- Daariec/Codex-translate-by-ArlenCode · 31 Oct 2026 · Daariec · 18 stars