Scaling LLM Pre-training with Vocabulary Curriculum
Modern language models rely on static vocabularies, fixed before pretraining,
in contrast to the adaptive vocabulary acquisition observed in human language
learning. To bridge this gap, we introduce vocabulary curriculum learning, an
approach that improves pretraining efficiency with log-linear scaling gains
relative to vocabulary size. Our method alternates between entropy-guided
vocabulary expansion and model optimization, enabling models to learn
transferable representations across diverse tokenization granularities. This
approach naturally gives rise to an optimal computation allocation pattern:
longer tokens capture predictable content, while shorter tokens focus on more
complex, harder-to-predict contexts. Experiments on small-scale GPT models
demonstrate improved scaling efficiency, reinforcing the effectiveness of
dynamic tokenization. We release our code to support further research and plan
to extend our experiments to larger models and diverse domains.
67be7f96b4ca41e2807a4fb0
Fangyuan Yu
claimed_verified
2025-02-26T08:26:07.778Z
64d98ef7a4839890b25eb78b
https://cdn-avatars.huggingface.co/v1/production/uploads/64d98ef7a4839890b25eb78b/215-CSVLl81z6CAq0ECWU.jpeg
Fangyuan Yu
user
Ksgk-fy
Scaling LLM Pre-training with Vocabulary Curriculum
Modern language models rely on static vocabularies, fixed before pretraining,
in contrast to the adaptive vocabulary acquisition observed in human language
learning. To bridge this gap, we introduce vocabulary curriculum learning, an
approach that improves pretraining efficiency with log-linear scaling gains
relative to vocabulary size. Our method alternates between entropy-guided
vocabulary expansion and model optimization, enabling models to learn
transferable representations across diverse tokenization granularities. This
approach naturally gives rise to an optimal computation allocation pattern:
longer tokens capture predictable content, while shorter tokens focus on more
complex, harder-to-predict contexts. Experiments on small-scale GPT models
demonstrate improved scaling efficiency, reinforcing the effectiveness of
dynamic tokenization. We release our code to support further research and plan
to extend our experiments to larger models and diverse domains.
Pioneering vocabulary curriculum learning enhances pretraining efficiency in language models by dynamically adjusting vocabulary and balancing tokenization granularity.
vocabulary curriculum learning entropy-guided vocabulary expansion model optimization transferable representations tokenization granularities