Reinforcement Learning on Pre-Training Data
The growing disparity between the exponential scaling of computational
resources and the finite growth of high-quality text data now constrains
conventional scaling approaches for large language models (LLMs). To address
this challenge, we introduce Reinforcement Learning on Pre-Training data
(RLPT), a new training-time scaling paradigm for optimizing LLMs. In contrast
to prior approaches that scale training primarily through supervised learning,
RLPT enables the policy to autonomously explore meaningful trajectories to
learn from pre-training data and improve its capability through reinforcement
learning (RL). While existing RL strategies such as reinforcement learning from
human feedback (RLHF) and reinforcement learning with verifiable rewards (RLVR)
rely on human annotation for reward construction, RLPT eliminates this
dependency by deriving reward signals directly from pre-training data.
Specifically, it adopts a next-segment reasoning objective, rewarding the
policy for accurately predicting subsequent text segments conditioned on the
preceding context. This formulation allows RL to be scaled on pre-training
data, encouraging the exploration of richer trajectories across broader
contexts and thereby fostering more generalizable reasoning skills. Extensive
experiments on both general-domain and mathematical reasoning benchmarks across
multiple models validate the effectiveness of RLPT. For example, when applied
to Qwen3-4B-Base, RLPT yields absolute improvements of 3.0, 5.1, 8.1,
6.0, 6.6, and 5.3 on MMLU, MMLU-Pro, GPQA-Diamond, KOR-Bench, AIME24, and
AIME25, respectively. The results further demonstrate favorable scaling
behavior, suggesting strong potential for continued gains with more compute. In
addition, RLPT provides a solid foundation, extending the reasoning boundaries
of LLMs and enhancing RLVR performance.
68d352680e215259d193b1fa
Siheng Li
claimed_verified
2025-09-24T13:53:23.476Z
66d45a8de5837f38ce3b73f7
https://cdn-avatars.huggingface.co/v1/production/uploads/66d45a8de5837f38ce3b73f7/3omslNRb8wV_c1xbrCmQC.jpeg
SihengLi
user
Siheng99
68d352680e215259d193b1fb
Kejiao Li
68d352680e215259d193b1fc
Zenan Xu
claimed_verified
2025-09-24T13:53:18.104Z
66dd5a7468f47ec63e502794
/avatars/6497fd124e6009777167feef2558f058.svg
xu
user
xavier-z
68d352680e215259d193b1fd
Guanhua Huang
68d352680e215259d193b1fe
Evander Yang
68d352680e215259d193b1ff
Kun Li
68d352680e215259d193b200
Haoyuan Wu
claimed_verified
2025-09-24T13:53:15.202Z
6445e7b1b272430bdbf64e80
/avatars/d3e59a3b488f8539966c944bb16f7b90.svg
Haoyuan WU
user
hywu
68d352680e215259d193b201
Jiajia Wu
68d352680e215259d193b202
Zihao Zheng
68d352680e215259d193b203
Chenchen Zhang
68d352680e215259d193b204
Kun Shi
68d352680e215259d193b205
Kyrierl Deng
68d352680e215259d193b206
Qi Yi
68d352680e215259d193b207
Ruibin Xiong
68d352680e215259d193b208
Tingqiang Xu
68d352680e215259d193b209
Yuhao Jiang
68d352680e215259d193b20a
Jianfeng Yan
68d352680e215259d193b20b
Yuyuan Zeng
68d352680e215259d193b20c
Guanghui Xu
68d352680e215259d193b20d
Jinbao Xue
68d352680e215259d193b20e
Zhijiang Xu
68d352680e215259d193b20f
Zheng Fang
68d352680e215259d193b210
Shuai Li
claimed_verified
2025-09-24T13:53:20.582Z
6525511b3f8d02205bf1e9ef
/avatars/7dfc3020337f3c403ecf133fa312b5d1.svg
Shuai Li
user
DiveBlue
68d352680e215259d193b211
Qibin Liu
68d352680e215259d193b212
Xiaoxue Li
68d352680e215259d193b213
Zhuoyu Li
68d352680e215259d193b214
Yangyu Tao
68d352680e215259d193b215
Fei Gao
68d352680e215259d193b216
Cheng Jiang
68d352680e215259d193b217
Bo Chao Wang
68d352680e215259d193b218
Kai Liu
68d352680e215259d193b219
Jianchen Zhu
68d352680e215259d193b21a
Wai Lam
68d352680e215259d193b21b
Wayyt Wang
68d352680e215259d193b21c
Bo Zhou
68d352680e215259d193b21d
Di Wang
Reinforcement Learning on Pre-Training Data
The growing disparity between the exponential scaling of computational
resources and the finite growth of high-quality text data now constrains
conventional scaling approaches for large language models (LLMs). To address
this challenge, we introduce Reinforcement Learning on Pre-Training data
(RLPT), a new training-time scaling paradigm for optimizing LLMs. In contrast
to prior approaches that scale training primarily through supervised learning,
RLPT enables the policy to autonomously explore meaningful trajectories to
learn from pre-training data and improve its capability through reinforcement
learning (RL). While existing RL strategies such as reinforcement learning from
human feedback (RLHF) and reinforcement learning with verifiable rewards (RLVR)
rely on human annotation for reward construction, RLPT eliminates this
dependency by deriving reward signals directly from pre-training data.
Specifically, it adopts a next-segment reasoning objective, rewarding the
policy for accurately predicting subsequent text segments conditioned on the
preceding context. This formulation allows RL to be scaled on pre-training
data, encouraging the exploration of richer trajectories across broader
contexts and thereby fostering more generalizable reasoning skills. Extensive
experiments on both general-domain and mathematical reasoning benchmarks across
multiple models validate the effectiveness of RLPT. For example, when applied
to Qwen3-4B-Base, RLPT yields absolute improvements of 3.0, 5.1, 8.1,
6.0, 6.6, and 5.3 on MMLU, MMLU-Pro, GPQA-Diamond, KOR-Bench, AIME24, and
AIME25, respectively. The results further demonstrate favorable scaling
behavior, suggesting strong potential for continued gains with more compute. In
addition, RLPT provides a solid foundation, extending the reasoning boundaries
of LLMs and enhancing RLVR performance.
Reinforcement Learning on Pre-Training data (RLPT) optimizes large language models by autonomously exploring meaningful trajectories in pre-training data, improving generalizable reasoning skills without human annotation.
Reinforcement Learning on Pre-Training data RLPT large language models LLMs reinforcement learning RL reinforcement learning from human feedback RLHF reinforcement learning with verifiable rewards RLVR next-segment reasoning objective MMLU MMLU-Pro GPQA-Diamond KOR-Bench AIME24 AIME25