Evaluating Tokenizer Performance of Large Language Models Across
  Official Indian Languages
Large Language Models (LLMs) based on transformer architectures have
revolutionized a variety of domains, with tokenization playing a pivotal role
in their pre-processing and fine-tuning stages. In multilingual models,
particularly those tailored for Indic languages, effective tokenization is
crucial for optimizing performance. This paper presents a comprehensive
evaluation of tokenizers used by 12 LLMs across all 22 official languages of
India, with a focus on comparing the efficiency of their tokenization
processes. We employed the Normalized Sequence Length (NSL) as a key metric in
our analysis. Our findings reveal that the SUTRA tokenizer outperforms all
other models, including several Indic-specific models, excelling in 14
languages. Notable insights include the SUTRA tokenizer's superior handling of
Indic languages, GPT-4o's advancement over its predecessor GPT-4 in processing
Indian languages, and the limited performance of Project Indus in certain
languages. This study underscores the critical importance of developing
targeted tokenization strategies for multilingual and Indic-centric models,
laying the groundwork for future improvements in tokenizer design to enhance
linguistic coverage and model efficiency.
673d599340b6ff437a9ae9d8
S. Tamang
claimed_verified
2024-11-20T10:08:33.948Z
65d3518c40e07f7dbf7ca772
/avatars/018e780ebeacc8440487d5f550574906.svg
SAGAR TAMANG
user
tamang0000
673d599340b6ff437a9ae9d9
D. J. Bora
Evaluating Tokenizer Performance of Large Language Models Across
  Official Indian Languages
Large Language Models (LLMs) based on transformer architectures have
revolutionized a variety of domains, with tokenization playing a pivotal role
in their pre-processing and fine-tuning stages. In multilingual models,
particularly those tailored for Indic languages, effective tokenization is
crucial for optimizing performance. This paper presents a comprehensive
evaluation of tokenizers used by 12 LLMs across all 22 official languages of
India, with a focus on comparing the efficiency of their tokenization
processes. We employed the Normalized Sequence Length (NSL) as a key metric in
our analysis. Our findings reveal that the SUTRA tokenizer outperforms all
other models, including several Indic-specific models, excelling in 14
languages. Notable insights include the SUTRA tokenizer's superior handling of
Indic languages, GPT-4o's advancement over its predecessor GPT-4 in processing
Indian languages, and the limited performance of Project Indus in certain
languages. This study underscores the critical importance of developing
targeted tokenization strategies for multilingual and Indic-centric models,
laying the groundwork for future improvements in tokenizer design to enhance
linguistic coverage and model efficiency.
A comprehensive evaluation finds SUTRA tokenizer excels for tokenizing Indic languages across various LLMs, surpassing competitors in handling multiple Indian languages.
Large Language Models transformer architectures tokenization multilingual models Indic languages Normalized Sequence Length SUTRA tokenizer GPT-4o Project Indus targeted tokenization strategies