method · first seen 14 Oct 2026
morphological tokenization
Also: morphology-aware tokenization, root-and-pattern tokenization
Tokenizing text along linguistic morpheme boundaries rather than purely frequency-driven segmentation such as BPE.
Mentions per week
PapersCodeDiscussion
Earliest sources
- Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in TagalogPapers · 14 Oct 2026 · Abdul Rehman Antall, Baidya Nath Saha
- What Do Tokenizer Comparisons Measure? Disentangling Granularity, Root Assignment, and Word-Level Readout across Amharic and HausaPapers · 10 Nov 2026 · Abdul Rehman Antall, Baidya Nath Saha
Latest sources
- What Do Tokenizer Comparisons Measure? Disentangling Granularity, Root Assignment, and Word-Level Readout across Amharic and HausaPapers · 10 Nov 2026 · Abdul Rehman Antall, Baidya Nath Saha
- Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in TagalogPapers · 14 Oct 2026 · Abdul Rehman Antall, Baidya Nath Saha