Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies
Research on scaling large language models (LLMs) has primarily focused on
model parameters and training data size, overlooking the role of vocabulary
size. % Intuitively, larger vocabularies enable more efficient tokenization by
representing sentences with fewer tokens, but they also increase the risk of
under-fitting representations for rare tokens. We investigate how vocabulary
size impacts LLM scaling laws by training models ranging from 33M to 3B
parameters on up to 500B characters with various vocabulary configurations. We
propose three complementary approaches for predicting the compute-optimal
vocabulary size: IsoFLOPs analysis, derivative estimation, and parametric fit
of the loss function. Our approaches converge on the same result that the
optimal vocabulary size depends on the available compute budget and that larger
models deserve larger vocabularies. However, most LLMs use too small vocabulary
sizes. For example, we predict that the optimal vocabulary size of Llama2-70B
should have been at least 216K, 7 times larger than its vocabulary of 32K. We
validate our predictions empirically by training models with 3B parameters
across different FLOPs budgets. Adopting our predicted optimal vocabulary size
consistently improves downstream performance over commonly used vocabulary
sizes. By increasing the vocabulary size from the conventional 32K to 43K, we
improve performance on ARC-Challenge from 29.1 to 32.0 with the same 2.3e21
FLOPs. Our work emphasizes the necessity of jointly considering model
parameters and vocabulary size for efficient scaling.
6699cc6534b724c13d591358
Chaofan Tao
6699cc6534b724c13d591359
Qian Liu
claimed_verified
2024-07-19T11:23:17.971Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
6699cc6534b724c13d59135a
Longxu Dou
claimed_verified
2024-07-19T11:23:15.966Z
6214e4ee1e35c843d42d1f88
https://cdn-avatars.huggingface.co/v1/production/uploads/6214e4ee1e35c843d42d1f88/fj-9wuIdPhvogh3BrcXTB.jpeg
Longxu Dou
user
dreamerdeo
6699cc6534b724c13d59135b
Niklas Muennighoff
6699cc6534b724c13d59135c
Zhongwei Wan
6699cc6534b724c13d59135d
Ping Luo
6699cc6534b724c13d59135e
Min Lin
6699cc6534b724c13d59135f
Ngai Wong
Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies
Research on scaling large language models (LLMs) has primarily focused on
model parameters and training data size, overlooking the role of vocabulary
size. % Intuitively, larger vocabularies enable more efficient tokenization by
representing sentences with fewer tokens, but they also increase the risk of
under-fitting representations for rare tokens. We investigate how vocabulary
size impacts LLM scaling laws by training models ranging from 33M to 3B
parameters on up to 500B characters with various vocabulary configurations. We
propose three complementary approaches for predicting the compute-optimal
vocabulary size: IsoFLOPs analysis, derivative estimation, and parametric fit
of the loss function. Our approaches converge on the same result that the
optimal vocabulary size depends on the available compute budget and that larger
models deserve larger vocabularies. However, most LLMs use too small vocabulary
sizes. For example, we predict that the optimal vocabulary size of Llama2-70B
should have been at least 216K, 7 times larger than its vocabulary of 32K. We
validate our predictions empirically by training models with 3B parameters
across different FLOPs budgets. Adopting our predicted optimal vocabulary size
consistently improves downstream performance over commonly used vocabulary
sizes. By increasing the vocabulary size from the conventional 32K to 43K, we
improve performance on ARC-Challenge from 29.1 to 32.0 with the same 2.3e21
FLOPs. Our work emphasizes the necessity of jointly considering model
parameters and vocabulary size for efficient scaling.
Investigating the impact of vocabulary size on the scaling of large language models reveals that larger vocabularies improve performance when considering compute budgets, and current models often use suboptimal vocabulary sizes.
large language models LLMs tokenization vocabulary size IsoFLOPs analysis derivative estimation parametric fit loss function compute-optimal vocabulary size ARC-Challenge