method · first seen 8 Oct 2026
vocabulary extension
Adding new tokens or merges to a pretrained tokenizer so it covers additional languages or scripts.
Mentions per week
PapersCodeDiscussion
Earliest sources
- Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued PretrainingPapers · 8 Oct 2026 · Aarjan Chaudhary
Latest sources
- Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued PretrainingPapers · 8 Oct 2026 · Aarjan Chaudhary