Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining
  Levels
Large Language Models (LLMs) have achieved remarkable success through
imitation learning on vast text corpora, but this paradigm creates a
training-generation gap and limits robust reasoning. Reinforcement learning
(RL) offers a more data-efficient solution capable of bridging this gap, yet
its application has been constrained by a critical data bottleneck: existing RL
datasets are orders of magnitude smaller and less diverse than web-scale
pre-training corpora. To address this, we introduce the Webscale-RL pipeline, a
scalable data engine that systematically converts large-scale pre-training
documents into millions of diverse, verifiable question-answer pairs for RL.
Using this pipeline, we construct the Webscale-RL dataset, containing 1.2
million examples across more than 9 domains. Our experiments show that the
model trained on this dataset significantly outperforms continual pretraining
and strong data refinement baselines across a suite of benchmarks. Notably, RL
training with our dataset proves substantially more efficient, achieving the
performance of continual pre-training with up to 100times fewer tokens. Our
work presents a viable path toward scaling RL to pre-training levels, enabling
more capable and efficient language models.
68e7f1de93680d7965e7c838
Zhepeng Cen
68e7f1de93680d7965e7c839
Haolin Chen
claimed_verified
2025-10-10T04:33:41.584Z
65e0c21ebd0c59bbf271004c
/avatars/3210f41fd433d798fd7857c943288625.svg
Haolin Chen
user
hlnchen
68e7f1de93680d7965e7c83a
Shiyu Wang
68e7f1de93680d7965e7c83b
Zuxin Liu
68e7f1de93680d7965e7c83c
Zhiwei Liu
68e7f1de93680d7965e7c83d
Ding Zhao
68e7f1de93680d7965e7c83e
Silvio Savarese
68e7f1de93680d7965e7c83f
Caiming Xiong
68e7f1de93680d7965e7c840
Huan Wang
68e7f1de93680d7965e7c841
Weiran Yao
Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining
  Levels
Large Language Models (LLMs) have achieved remarkable success through
imitation learning on vast text corpora, but this paradigm creates a
training-generation gap and limits robust reasoning. Reinforcement learning
(RL) offers a more data-efficient solution capable of bridging this gap, yet
its application has been constrained by a critical data bottleneck: existing RL
datasets are orders of magnitude smaller and less diverse than web-scale
pre-training corpora. To address this, we introduce the Webscale-RL pipeline, a
scalable data engine that systematically converts large-scale pre-training
documents into millions of diverse, verifiable question-answer pairs for RL.
Using this pipeline, we construct the Webscale-RL dataset, containing 1.2
million examples across more than 9 domains. Our experiments show that the
model trained on this dataset significantly outperforms continual pretraining
and strong data refinement baselines across a suite of benchmarks. Notably, RL
training with our dataset proves substantially more efficient, achieving the
performance of continual pre-training with up to 100times fewer tokens. Our
work presents a viable path toward scaling RL to pre-training levels, enabling
more capable and efficient language models.
A scalable data engine converts large-scale pre-training documents into diverse question-answer pairs for reinforcement learning, significantly improving model performance and efficiency.
Large Language Models imitation learning reinforcement learning Webscale-RL pipeline Webscale-RL dataset continual pretraining data refinement benchmarks