Exploring the Limit of Outcome Reward for Learning Mathematical
  Reasoning
Reasoning abilities, especially those for solving complex math problems, are
crucial components of general intelligence. Recent advances by proprietary
companies, such as o-series models of OpenAI, have made remarkable progress on
reasoning tasks. However, the complete technical details remain unrevealed, and
the techniques that are believed certainly to be adopted are only reinforcement
learning (RL) and the long chain of thoughts. This paper proposes a new RL
framework, termed OREAL, to pursue the performance limit that can be achieved
through Outcome REwArd-based reinforcement
Learning for mathematical reasoning tasks, where only binary outcome
rewards are easily accessible. We theoretically prove that behavior cloning on
positive trajectories from best-of-N (BoN) sampling is sufficient to learn the
KL-regularized optimal policy in binary feedback environments. This formulation
further implies that the rewards of negative samples should be reshaped to
ensure the gradient consistency between positive and negative samples. To
alleviate the long-existing difficulties brought by sparse rewards in RL, which
are even exacerbated by the partial correctness of the long chain of thought
for reasoning tasks, we further apply a token-level reward model to sample
important tokens in reasoning trajectories for learning. With OREAL, for the
first time, a 7B model can obtain 94.0 pass@1 accuracy on MATH-500 through RL,
being on par with 32B models. OREAL-32B also surpasses previous 32B models
trained by distillation with 95.0 pass@1 accuracy on MATH-500. Our
investigation also indicates the importance of initial policy models and
training queries for RL. Code, models, and data will be released to benefit
future researchhttps://github.com/InternLM/OREAL.
67aacd7e078cdf445284f9f6
Chengqi Lyu
67aacd7e078cdf445284f9f7
Songyang Gao
admin_assigned
2025-02-11T15:56:04.271Z
650ab54e23196fb2d86b486b
/avatars/e0506393589695b553ec9ee3fe99b93a.svg
SongYang Gao
user
Wizardcoast
67aacd7e078cdf445284f9f8
Yuzhe Gu
claimed_verified
2025-02-18T16:40:46.962Z
6601196cc91ba4c08ad6e270
https://cdn-avatars.huggingface.co/v1/production/uploads/6601196cc91ba4c08ad6e270/X2YPNzUOQXBz5Gv-xR9LW.jpeg
yuzhe gu
user
vanilla1116
67aacd7e078cdf445284f9f9
Wenwei Zhang
claimed_verified
2025-02-11T07:54:40.279Z
64e8505321540e1da3226b54
/avatars/18958b8406d1ce492b54c1c839f18c54.svg
Wenwei Zhang
user
ZwwWayne
67aacd7e078cdf445284f9fa
Jianfei Gao
admin_assigned
2025-02-11T15:56:31.173Z
64070c5c4dc5f2846c925e93
/avatars/ac2d7c1cd4ecccd6a88b85767c963ec7.svg
Gao Jianfei
user
pppppM
67aacd7e078cdf445284f9fb
Kuikun Liu
claimed_verified
2025-07-23T08:38:41.625Z
63fd691794cc8f815d50c112
/avatars/87305d1cbfcc717e910ccdfaf0568f80.svg
liu
user
Harold-lkk
67aacd7e078cdf445284f9fc
Ziyi Wang
67aacd7e078cdf445284f9fd
Shuaibin Li
67aacd7e078cdf445284f9fe
Qian Zhao
67aacd7e078cdf445284f9ff
Haian Huang
67aacd7e078cdf445284fa00
Weihan Cao
67aacd7e078cdf445284fa01
Jiangning Liu
67aacd7e078cdf445284fa02
Hongwei Liu
67aacd7e078cdf445284fa03
Junnan Liu
claimed_verified
2025-05-27T07:56:53.050Z
643d26979347842571bc9613
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/3heFf7h3jbhhJWJ4JfGfh.jpeg
Junnan Liu
user
jnanliu
67aacd7e078cdf445284fa04
Songyang Zhang
claimed_verified
2025-02-11T07:54:37.733Z
630716d11801ecc7d2595021
/avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
Songyang Zhang
user
zsytony
67aacd7e078cdf445284fa05
Dahua Lin
admin_assigned
2025-02-19T16:16:07.907Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
67aacd7e078cdf445284fa06
Kai Chen
Exploring the Limit of Outcome Reward for Learning Mathematical
  Reasoning
Reasoning abilities, especially those for solving complex math problems, are
crucial components of general intelligence. Recent advances by proprietary
companies, such as o-series models of OpenAI, have made remarkable progress on
reasoning tasks. However, the complete technical details remain unrevealed, and
the techniques that are believed certainly to be adopted are only reinforcement
learning (RL) and the long chain of thoughts. This paper proposes a new RL
framework, termed OREAL, to pursue the performance limit that can be achieved
through Outcome REwArd-based reinforcement
Learning for mathematical reasoning tasks, where only binary outcome
rewards are easily accessible. We theoretically prove that behavior cloning on
positive trajectories from best-of-N (BoN) sampling is sufficient to learn the
KL-regularized optimal policy in binary feedback environments. This formulation
further implies that the rewards of negative samples should be reshaped to
ensure the gradient consistency between positive and negative samples. To
alleviate the long-existing difficulties brought by sparse rewards in RL, which
are even exacerbated by the partial correctness of the long chain of thought
for reasoning tasks, we further apply a token-level reward model to sample
important tokens in reasoning trajectories for learning. With OREAL, for the
first time, a 7B model can obtain 94.0 pass@1 accuracy on MATH-500 through RL,
being on par with 32B models. OREAL-32B also surpasses previous 32B models
trained by distillation with 95.0 pass@1 accuracy on MATH-500. Our
investigation also indicates the importance of initial policy models and
training queries for RL. Code, models, and data will be released to benefit
future researchhttps://github.com/InternLM/OREAL.
A novel reinforcement learning framework, OREAL, using outcome rewards achieves high accuracy in mathematical reasoning tasks with smaller models compared to larger models trained using distillation.
reinforcement learning OREAL Outcome REwArd-based reinforcement Learning binary outcome rewards behavior cloning best-of-N KL-regularized optimal policy token-level reward model MATH-500 pass@1 accuracy