trends.barnas.net

phenomenon · first seen 17 Oct 2026

tokenizer fertility

The average number of subword tokens produced per word, indicating tokenization efficiency across languages.

Mentions per week

0219 May11 Aug10 NovPapersCodeDiscussion

PapersCodeDiscussion

Earliest sources

Latest sources

Mentioned together with

byte-fallback tokenizationlow-resource language modelingmultilingual language model evaluationlow-rank adaptationbits per byteSentencePiece