BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale
  Pretraining
Recent advances in large language model (LLM) pretraining have shown that
simply scaling data quantity eventually leads to diminishing returns, hitting a
data wall. In response, the use of synthetic data for pretraining has emerged
as a promising paradigm for pushing the frontier of performance. Despite this,
the factors affecting synthetic data quality remain poorly understood. In this
work, we introduce BeyondWeb, a synthetic data generation framework that
produces high-quality synthetic data for pretraining. BeyondWeb significantly
extends the capabilities of traditional web-scale datasets, outperforming
state-of-the-art synthetic pretraining datasets such as Cosmopedia and
Nemotron-CC's high-quality synthetic subset (Nemotron-Synth) by up to 5.1
percentage points (pp) and 2.6pp, respectively, when averaged across a suite of
14 benchmark evaluations. It delivers up to 7.7x faster training than open web
data and 2.7x faster than Nemotron-Synth. Remarkably, a 3B model trained for
180B tokens on BeyondWeb outperforms an 8B model trained for the same token
budget on Cosmopedia. We also present several insights from BeyondWeb on
synthetic data for pretraining: what drives its benefits, which data to
rephrase and how, and the impact of model size and family on data quality.
Overall, our work shows that there's no silver bullet for generating
high-quality synthetic pretraining data. The best outcomes require jointly
optimizing many factors, a challenging task that requires rigorous science and
practical expertise. Naive approaches can yield modest improvements,
potentially at great cost, while well-executed methods can yield transformative
improvements, as exemplified by BeyondWeb.
68a32d11b65388761d07436f
Pratyush Maini
claimed_verified
2025-08-19T08:05:02.279Z
632c826d1d303f5f9acf5917
/avatars/defca406f2c4f027a7b1be9a5ebbbd01.svg
Pratyush Maini
user
pratyushmaini
68a32d11b65388761d074370
Vineeth Dorna
68a32d11b65388761d074371
Parth Doshi
68a32d11b65388761d074372
Aldo Carranza
68a32d11b65388761d074373
Fan Pan
68a32d11b65388761d074374
Jack Urbanek
68a32d11b65388761d074375
Paul Burstein
68a32d11b65388761d074376
Alex Fang
68a32d11b65388761d074377
Alvin Deng
68a32d11b65388761d074378
Amro Abbas
68a32d11b65388761d074379
Brett Larsen
68a32d11b65388761d07437a
Cody Blakeney
68a32d11b65388761d07437b
Charvi Bannur
68a32d11b65388761d07437c
Christina Baek
68a32d11b65388761d07437d
Darren Teh
68a32d11b65388761d07437e
David Schwab
68a32d11b65388761d07437f
Haakon Mongstad
68a32d11b65388761d074380
Haoli Yin
68a32d11b65388761d074381
Josh Wills
68a32d11b65388761d074382
Kaleigh Mentzer
68a32d11b65388761d074383
Luke Merrick
68a32d11b65388761d074384
Ricardo Monti
claimed_verified
2025-08-19T08:05:00.088Z
67aaba8191e6f5eb54751e30
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/-LEpeO606QKWbUi2gLH6f.png
Ricardo Monti
user
ricardomonti08
68a32d11b65388761d074385
Rishabh Adiga
68a32d11b65388761d074386
Siddharth Joshi
68a32d11b65388761d074387
Spandan Das
68a32d11b65388761d074388
Zhengping Wang
68a32d11b65388761d074389
Bogdan Gaza
68a32d11b65388761d07438a
Ari Morcos
68a32d11b65388761d07438b
Matthew Leavitt
BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale
  Pretraining
Recent advances in large language model (LLM) pretraining have shown that
simply scaling data quantity eventually leads to diminishing returns, hitting a
data wall. In response, the use of synthetic data for pretraining has emerged
as a promising paradigm for pushing the frontier of performance. Despite this,
the factors affecting synthetic data quality remain poorly understood. In this
work, we introduce BeyondWeb, a synthetic data generation framework that
produces high-quality synthetic data for pretraining. BeyondWeb significantly
extends the capabilities of traditional web-scale datasets, outperforming
state-of-the-art synthetic pretraining datasets such as Cosmopedia and
Nemotron-CC's high-quality synthetic subset (Nemotron-Synth) by up to 5.1
percentage points (pp) and 2.6pp, respectively, when averaged across a suite of
14 benchmark evaluations. It delivers up to 7.7x faster training than open web
data and 2.7x faster than Nemotron-Synth. Remarkably, a 3B model trained for
180B tokens on BeyondWeb outperforms an 8B model trained for the same token
budget on Cosmopedia. We also present several insights from BeyondWeb on
synthetic data for pretraining: what drives its benefits, which data to
rephrase and how, and the impact of model size and family on data quality.
Overall, our work shows that there's no silver bullet for generating
high-quality synthetic pretraining data. The best outcomes require jointly
optimizing many factors, a challenging task that requires rigorous science and
practical expertise. Naive approaches can yield modest improvements,
potentially at great cost, while well-executed methods can yield transformative
improvements, as exemplified by BeyondWeb.
BeyondWeb, a synthetic data generation framework, outperforms existing datasets and accelerates training for large language models by optimizing multiple factors.
large language model LLM synthetic data pretraining data wall BeyondWeb Cosmopedia Nemotron-Synth benchmark evaluations token budget model size model family