method · first seen 8 Oct 2026
pre-tokenization
Splitting raw text into chunks with a regular expression before subword tokenization is applied.
Mentions per week
PapersCodeDiscussion
Earliest sources
- Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued PretrainingPapers · 8 Oct 2026 · Aarjan Chaudhary
- FIDEL: Abugida-Aware Pre-tokenization for Ethiopic LanguagesPapers · 15 Nov 2026 · Gebreslassie Teklu Reda, Yuval Pinter
Latest sources
- FIDEL: Abugida-Aware Pre-tokenization for Ethiopic LanguagesPapers · 15 Nov 2026 · Gebreslassie Teklu Reda, Yuval Pinter
- Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued PretrainingPapers · 8 Oct 2026 · Aarjan Chaudhary