method · first seen 15 Nov 2026
subword tokenization
Also: subword tokenizer
Splitting text into learned subword units that serve as the input vocabulary for language models.
Mentions per week
PapersCodeDiscussion
Earliest sources
- FIDEL: Abugida-Aware Pre-tokenization for Ethiopic LanguagesPapers · 15 Nov 2026 · Gebreslassie Teklu Reda, Yuval Pinter
Latest sources
- FIDEL: Abugida-Aware Pre-tokenization for Ethiopic LanguagesPapers · 15 Nov 2026 · Gebreslassie Teklu Reda, Yuval Pinter