Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling
Tokenization is a fundamental component of large language models (LLMs), yet
its influence on model scaling and performance is not fully explored. In this
paper, we introduce Over-Tokenized Transformers, a novel framework that
decouples input and output vocabularies to improve language modeling
performance. Specifically, our approach scales up input vocabularies to
leverage multi-gram tokens. Through extensive experiments, we uncover a
log-linear relationship between input vocabulary size and training loss,
demonstrating that larger input vocabularies consistently enhance model
performance, regardless of model size. Using a large input vocabulary, we
achieve performance comparable to double-sized baselines with no additional
cost. Our findings highlight the importance of tokenization in scaling laws and
provide practical insight for tokenizer design, paving the way for more
efficient and powerful LLMs.
6799b345a66ae6b357bef986
Hongzhi Huang
claimed_verified
2025-01-29T17:16:08.642Z
66fbbed8a0154e0b6498293d
/avatars/5c4202f13cc5af6424154fae293fad52.svg
Huang Hongzhi
user
xyzed
6799b345a66ae6b357bef987
Defa Zhu
claimed_verified
2025-01-29T17:16:06.916Z
667505f4361b960c79e35486
/avatars/d352639c520075220f6abaae23c39376.svg
Defa Zhu
user
mathfinder
6799b345a66ae6b357bef988
Banggu Wu
admin_assigned
2025-01-29T17:21:12.737Z
64a3b0fa565496b629879293
/avatars/9d5de6cc4a01052ea971701f72bd3489.svg
wubanggu
user
banggu
6799b345a66ae6b357bef989
Yutao Zeng
claimed_verified
2025-01-29T17:16:10.495Z
6371128eafbe42caa5a5222b
/avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
Yutao Zeng
user
Taoer
6799b345a66ae6b357bef98a
Ya Wang
6799b345a66ae6b357bef98b
Qiyang Min
admin_assigned
2025-01-29T17:21:23.469Z
645507b89d37c3fb33279fe3
/avatars/ec1122137c49204ab968182d1f726c35.svg
min
user
qiyang-attn
6799b345a66ae6b357bef98c
Xun Zhou
admin_assigned
2025-01-29T17:21:37.188Z
62533db4a06ec75172eeabe7
/avatars/b1a4dad90afae5c00df97233a97777db.svg
xunzhou
user
xunzhou
Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling
Tokenization is a fundamental component of large language models (LLMs), yet
its influence on model scaling and performance is not fully explored. In this
paper, we introduce Over-Tokenized Transformers, a novel framework that
decouples input and output vocabularies to improve language modeling
performance. Specifically, our approach scales up input vocabularies to
leverage multi-gram tokens. Through extensive experiments, we uncover a
log-linear relationship between input vocabulary size and training loss,
demonstrating that larger input vocabularies consistently enhance model
performance, regardless of model size. Using a large input vocabulary, we
achieve performance comparable to double-sized baselines with no additional
cost. Our findings highlight the importance of tokenization in scaling laws and
provide practical insight for tokenizer design, paving the way for more
efficient and powerful LLMs.
Over-Tokenized Transformers improve language modeling by decoupling input and output vocabularies, leading to better performance with larger input vocabularies.
Over-Tokenized Transformers multi-gram tokens input vocabulary size training loss language modeling tokenization scaling laws