trends.barnas.net

method · first seen 15 Nov 2026

subword tokenization

Also: subword tokenizer

Splitting text into learned subword units that serve as the input vocabulary for language models.

Mentions per week

0119 May11 Aug10 NovPapersCodeDiscussion

PapersCodeDiscussion

Earliest sources

Latest sources

Mentioned together with

pre-tokenization