Explaining and Mitigating Crosslingual Tokenizer Inequities
Catherine Arnett, Tyler Chang, Stella Biderman, Benjamin Bergen
The number of tokens it takes to encode parallel text in different languages is known to vary. These disparities are called *token premiums*.Having high token premiums leads to less throughput during training and increases costs at inference. In this paper, we find that even after controlling for dataset size, vocabulary size, and data content, monolingual tokenizers exhibit a wide range of token premiums across languages. To understand the cross-linguistic differences that affect these token premiums, we train a suite of almost 7,000 comparable monolingual tokenizers for 98 languages, manipulating tokenizer type (BPE,  Unigram, and SuperBPE), vocabulary size, and dataset size. We measure token premiums and test for a relationship between factors such as data similarity (between tokenizer training and evaluation), vocabulary size, and pre-tokenization. We also investigate the role of language-specific features such as writing system and word length. We find that similarity between training and test data does not impact token premiums, but vocabulary size and pre-tokenization do. While simply increasing vocabulary size does not lead to reduced token premium effects, we can determine an "optimal'' vocabulary size for each language to achieve significantly reduced token premium effects. We also train superword tokenizers which allow merges over whitespaces, and we find that they both reduce token premium effects and improve compression overall. Thus, intervening on either vocabulary size and the pre-tokenizer significantly reduces crosslingual token premium effects.