RLP: Reinforcement as a Pretraining Objective
The dominant paradigm for training large reasoning models starts with
pre-training using next-token prediction loss on vast amounts of data.
Reinforcement learning, while powerful in scaling reasoning, is introduced only
as the very last phase of post-training, preceded by supervised fine-tuning.
While dominant, is this an optimal way of training? In this paper, we present
RLP, an information-driven reinforcement pretraining objective, that brings the
core spirit of reinforcement learning -- exploration -- to the last phase of
pretraining. The key idea is to treat chain-of-thought as an exploratory
action, with rewards computed based on the information gain it provides for
predicting future tokens. This training objective essentially encourages the
model to think for itself before predicting what comes next, thus teaching an
independent thinking behavior earlier in the pretraining. More concretely, the
reward signal measures the increase in log-likelihood of the next token when
conditioning on both context and a sampled reasoning chain, compared to
conditioning on context alone. This approach yields a verifier-free dense
reward signal, allowing for efficient training for the full document stream
during pretraining. Specifically, RLP reframes reinforcement learning for
reasoning as a pretraining objective on ordinary text, bridging the gap between
next-token prediction and the emergence of useful chain-of-thought reasoning.
Pretraining with RLP on Qwen3-1.7B-Base lifts the overall average across an
eight-benchmark math-and-science suite by 19%. With identical post-training,
the gains compound, with the largest improvements on reasoning-heavy tasks such
as AIME25 and MMLU-Pro. Applying RLP to the hybrid Nemotron-Nano-12B-v2
increases the overall average from 42.81% to 61.32% and raises the average on
scientific reasoning by 23%, demonstrating scalability across architectures and
model sizes.
68df26d8df49fb0df1e03ba8
Ali Hatamizadeh
claimed_verified
2025-10-13T10:10:34.346Z
64414b62603214724ebd2636
https://cdn-avatars.huggingface.co/v1/production/uploads/64414b62603214724ebd2636/x9JVcJRZKZE7hdEII1JRR.jpeg
Ali
user
ahatamiz
68df26d8df49fb0df1e03ba9
Syeda Nahida Akter
claimed_verified
2025-10-03T12:33:13.536Z
6338dd1776421c0543150467
/avatars/4539dcec644e40be33f4a0d419fa66cb.svg
Syeda Nahida Akter
user
SieraL
68df26d8df49fb0df1e03baa
Shrimai Prabhumoye
68df26d8df49fb0df1e03bab
Jan Kautz
68df26d8df49fb0df1e03bac
Mostofa Patwary
68df26d8df49fb0df1e03bad
Mohammad Shoeybi
68df26d8df49fb0df1e03bae
Bryan Catanzaro
68df26d8df49fb0df1e03baf
Yejin Choi
RLP: Reinforcement as a Pretraining Objective
The dominant paradigm for training large reasoning models starts with
pre-training using next-token prediction loss on vast amounts of data.
Reinforcement learning, while powerful in scaling reasoning, is introduced only
as the very last phase of post-training, preceded by supervised fine-tuning.
While dominant, is this an optimal way of training? In this paper, we present
RLP, an information-driven reinforcement pretraining objective, that brings the
core spirit of reinforcement learning -- exploration -- to the last phase of
pretraining. The key idea is to treat chain-of-thought as an exploratory
action, with rewards computed based on the information gain it provides for
predicting future tokens. This training objective essentially encourages the
model to think for itself before predicting what comes next, thus teaching an
independent thinking behavior earlier in the pretraining. More concretely, the
reward signal measures the increase in log-likelihood of the next token when
conditioning on both context and a sampled reasoning chain, compared to
conditioning on context alone. This approach yields a verifier-free dense
reward signal, allowing for efficient training for the full document stream
during pretraining. Specifically, RLP reframes reinforcement learning for
reasoning as a pretraining objective on ordinary text, bridging the gap between
next-token prediction and the emergence of useful chain-of-thought reasoning.
Pretraining with RLP on Qwen3-1.7B-Base lifts the overall average across an
eight-benchmark math-and-science suite by 19%. With identical post-training,
the gains compound, with the largest improvements on reasoning-heavy tasks such
as AIME25 and MMLU-Pro. Applying RLP to the hybrid Nemotron-Nano-12B-v2
increases the overall average from 42.81% to 61.32% and raises the average on
scientific reasoning by 23%, demonstrating scalability across architectures and
model sizes.
RLP, an information-driven reinforcement pretraining objective, enhances reasoning models by integrating exploration into pretraining, leading to significant performance improvements across various benchmarks.
reinforcement learning pre-training next-token prediction chain-of-thought information gain log-likelihood dense reward signal Qwen3-1.7B-Base AIME25 MMLU-Pro Nemotron-Nano-12B-v2 scientific reasoning