DataDecide: How to Predict Best Pretraining Data with Small Experiments
Because large language models are expensive to pretrain on different
datasets, using smaller-scale experiments to decide on data is crucial for
reducing costs. Which benchmarks and methods of making decisions from observed
performance at small scale most accurately predict the datasets that yield the
best large models? To empower open exploration of this question, we release
models, data, and evaluations in DataDecide -- the most extensive open suite of
models over differences in data and scale. We conduct controlled pretraining
experiments across 25 corpora with differing sources, deduplication, and
filtering up to 100B tokens, model sizes up to 1B parameters, and 3 random
seeds. We find that the ranking of models at a single, small size (e.g., 150M
parameters) is a strong baseline for predicting best models at our larger
target scale (1B) (~80% of com parisons correct). No scaling law methods among
8 baselines exceed the compute-decision frontier of single-scale predictions,
but DataDecide can measure improvement in future scaling laws. We also identify
that using continuous likelihood metrics as proxies in small experiments makes
benchmarks including MMLU, ARC, HellaSwag, MBPP, and HumanEval >80% predictable
at the target 1B scale with just 0.01% of the compute.
67fffbd0fd5600e97f8deeff
Ian Magnusson
67fffbd0fd5600e97f8def00
Nguyen Tai
claimed_verified
2025-04-22T10:14:10.388Z
6359a17e3568d26d3f0c4f23
https://cdn-avatars.huggingface.co/v1/production/uploads/6359a17e3568d26d3f0c4f23/l523tmkBiZ9a-lGUMz8g8.jpeg
Tai Nguyen
user
taidng
67fffbd0fd5600e97f8def01
Ben Bogin
67fffbd0fd5600e97f8def02
David Heineman
claimed_verified
2025-04-22T10:14:07.187Z
6105c8136377c5f8ef2494b1
https://cdn-avatars.huggingface.co/v1/production/uploads/6105c8136377c5f8ef2494b1/mlifkrHYMhXlXSa9DFbDh.jpeg
David Heineman
user
davidheineman
67fffbd0fd5600e97f8def03
Jena D. Hwang
claimed_verified
2025-06-05T07:29:23.920Z
65c157eaf9957360824b0d87
https://cdn-avatars.huggingface.co/v1/production/uploads/65c157eaf9957360824b0d87/c1NWTKsFLATJLR7W6Kmqz.jpeg
jena hwang
user
jenah
67fffbd0fd5600e97f8def04
Luca Soldaini
67fffbd0fd5600e97f8def05
Akshita Bhagia
claimed_verified
2025-04-18T09:40:20.663Z
6282937a9624ab5f75c977e1
https://cdn-avatars.huggingface.co/v1/production/uploads/6282937a9624ab5f75c977e1/ZOysQbGzEzmrN_SaTQgOP.jpeg
Akshita Bhagia
user
akshitab
67fffbd0fd5600e97f8def06
Jiacheng Liu
67fffbd0fd5600e97f8def07
Dirk Groeneveld
67fffbd0fd5600e97f8def08
Oyvind Tafjord
67fffbd0fd5600e97f8def09
Noah A. Smith
67fffbd0fd5600e97f8def0a
Pang Wei Koh
67fffbd0fd5600e97f8def0b
Jesse Dodge
DataDecide: How to Predict Best Pretraining Data with Small Experiments
Because large language models are expensive to pretrain on different
datasets, using smaller-scale experiments to decide on data is crucial for
reducing costs. Which benchmarks and methods of making decisions from observed
performance at small scale most accurately predict the datasets that yield the
best large models? To empower open exploration of this question, we release
models, data, and evaluations in DataDecide -- the most extensive open suite of
models over differences in data and scale. We conduct controlled pretraining
experiments across 25 corpora with differing sources, deduplication, and
filtering up to 100B tokens, model sizes up to 1B parameters, and 3 random
seeds. We find that the ranking of models at a single, small size (e.g., 150M
parameters) is a strong baseline for predicting best models at our larger
target scale (1B) (~80% of com parisons correct). No scaling law methods among
8 baselines exceed the compute-decision frontier of single-scale predictions,
but DataDecide can measure improvement in future scaling laws. We also identify
that using continuous likelihood metrics as proxies in small experiments makes
benchmarks including MMLU, ARC, HellaSwag, MBPP, and HumanEval >80% predictable
at the target 1B scale with just 0.01% of the compute.
Small-scale experiments accurately predict the best large language models when using continuous likelihood metrics as proxies.
large language models pretraining benchmarks scaling laws DataDecide MMLU ARC HellaSwag MBPP HumanEval continuous likelihood metrics