CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for
  Language Model Pre-training
Pre-training datasets are typically collected from web content and lack
inherent domain divisions. For instance, widely used datasets like Common Crawl
do not include explicit domain labels, while manually curating labeled datasets
such as The Pile is labor-intensive. Consequently, identifying an optimal
pre-training data mixture remains a challenging problem, despite its
significant benefits for pre-training performance. To address these challenges,
we propose CLustering-based Iterative Data Mixture Bootstrapping (CLIMB), an
automated framework that discovers, evaluates, and refines data mixtures in a
pre-training setting. Specifically, CLIMB embeds and clusters large-scale
datasets in a semantic space and then iteratively searches for optimal mixtures
using a smaller proxy model and a predictor. When continuously trained on 400B
tokens with this mixture, our 1B model exceeds the state-of-the-art
Llama-3.2-1B by 2.0%. Moreover, we observe that optimizing for a specific
domain (e.g., Social Sciences) yields a 5% improvement over random sampling.
Finally, we introduce ClimbLab, a filtered 1.2-trillion-token corpus with 20
clusters as a research playground, and ClimbMix, a compact yet powerful
400-billion-token dataset designed for efficient pre-training that delivers
superior performance under an equal token budget. We analyze the final data
mixture, elucidating the characteristics of an optimal data mixture. Our data
is available at: https://research.nvidia.com/labs/lpr/climb/
6801d661ed5fc062197db592
Shizhe Diao
claimed_verified
2025-04-18T09:38:46.555Z
633bd54b00732349209a18fe
/avatars/150aa5b8d9bcca24366402932bf2d049.svg
Shizhe Diao
user
shizhediao
6801d661ed5fc062197db593
Yu Yang
6801d661ed5fc062197db594
Yonggan Fu
6801d661ed5fc062197db595
Xin Dong
6801d661ed5fc062197db596
Dan Su
claimed_verified
2025-06-05T07:29:21.716Z
67e34615c15564fd97f697ce
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/BNXjsEA5rGn7uFoFuVkB6.png
Dan Su
user
sudandandansu1
6801d661ed5fc062197db597
Markus Kliegl
6801d661ed5fc062197db598
Zijia Chen
6801d661ed5fc062197db599
Peter Belcak
claimed_verified
2025-07-07T11:26:43.322Z
63e8cccddd2c4effdd6283cf
/avatars/b289d4dda0aafd60af3f14e19837b69c.svg
Peter Belcak
user
pbelcak
6801d661ed5fc062197db59a
Yoshi Suhara
6801d661ed5fc062197db59b
Hongxu Yin
6801d661ed5fc062197db59c
Mostofa Patwary
6801d661ed5fc062197db59d
Yingyan
6801d661ed5fc062197db59e
Lin
6801d661ed5fc062197db59f
Jan Kautz
6801d661ed5fc062197db5a0
Pavlo Molchanov
claimed_verified
2025-04-20T15:03:26.583Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
user
pmolchanov
CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for
  Language Model Pre-training
Pre-training datasets are typically collected from web content and lack
inherent domain divisions. For instance, widely used datasets like Common Crawl
do not include explicit domain labels, while manually curating labeled datasets
such as The Pile is labor-intensive. Consequently, identifying an optimal
pre-training data mixture remains a challenging problem, despite its
significant benefits for pre-training performance. To address these challenges,
we propose CLustering-based Iterative Data Mixture Bootstrapping (CLIMB), an
automated framework that discovers, evaluates, and refines data mixtures in a
pre-training setting. Specifically, CLIMB embeds and clusters large-scale
datasets in a semantic space and then iteratively searches for optimal mixtures
using a smaller proxy model and a predictor. When continuously trained on 400B
tokens with this mixture, our 1B model exceeds the state-of-the-art
Llama-3.2-1B by 2.0%. Moreover, we observe that optimizing for a specific
domain (e.g., Social Sciences) yields a 5% improvement over random sampling.
Finally, we introduce ClimbLab, a filtered 1.2-trillion-token corpus with 20
clusters as a research playground, and ClimbMix, a compact yet powerful
400-billion-token dataset designed for efficient pre-training that delivers
superior performance under an equal token budget. We analyze the final data
mixture, elucidating the characteristics of an optimal data mixture. Our data
is available at: https://research.nvidia.com/labs/lpr/climb/
CLIMB semantic space proxy model predictor ClimbLab ClimbMix