ExGRPO: Learning to Reason from Experience
Reinforcement learning from verifiable rewards (RLVR) is an emerging paradigm
for improving the reasoning ability of large language models. However, standard
on-policy training discards rollout experiences after a single update, leading
to computational inefficiency and instability. While prior work on RL has
highlighted the benefits of reusing past experience, the role of experience
characteristics in shaping learning dynamics of large reasoning models remains
underexplored. In this paper, we are the first to investigate what makes a
reasoning experience valuable and identify rollout correctness and entropy as
effective indicators of experience value. Based on these insights, we propose
ExGRPO (Experiential Group Relative Policy Optimization), a framework that
organizes and prioritizes valuable experiences, and employs a mixed-policy
objective to balance exploration with experience exploitation. Experiments on
five backbone models (1.5B-8B parameters) show that ExGRPO consistently
improves reasoning performance on mathematical/general benchmarks, with an
average gain of +3.5/7.6 points over on-policy RLVR. Moreover, ExGRPO
stabilizes training on both stronger and weaker models where on-policy methods
fail. These results highlight principled experience management as a key
ingredient for efficient and scalable RLVR.
68df3b51df49fb0df1e03cd2
Runzhe Zhan
claimed_verified
2025-10-03T12:29:46.232Z
62495cb96ee7ee6b646db130
https://cdn-avatars.huggingface.co/v1/production/uploads/62495cb96ee7ee6b646db130/UwBXmvcMq7LMvBWUw0xo3.jpeg
Runzhe Zhan
user
rzzhan
68df3b51df49fb0df1e03cd3
Yafu Li
68df3b51df49fb0df1e03cd4
Zhi Wang
68df3b51df49fb0df1e03cd5
Xiaoye Qu
68df3b51df49fb0df1e03cd6
Dongrui Liu
68df3b51df49fb0df1e03cd7
Jing Shao
68df3b51df49fb0df1e03cd8
Derek F. Wong
68df3b51df49fb0df1e03cd9
Yu Cheng
ExGRPO: Learning to Reason from Experience
Reinforcement learning from verifiable rewards (RLVR) is an emerging paradigm
for improving the reasoning ability of large language models. However, standard
on-policy training discards rollout experiences after a single update, leading
to computational inefficiency and instability. While prior work on RL has
highlighted the benefits of reusing past experience, the role of experience
characteristics in shaping learning dynamics of large reasoning models remains
underexplored. In this paper, we are the first to investigate what makes a
reasoning experience valuable and identify rollout correctness and entropy as
effective indicators of experience value. Based on these insights, we propose
ExGRPO (Experiential Group Relative Policy Optimization), a framework that
organizes and prioritizes valuable experiences, and employs a mixed-policy
objective to balance exploration with experience exploitation. Experiments on
five backbone models (1.5B-8B parameters) show that ExGRPO consistently
improves reasoning performance on mathematical/general benchmarks, with an
average gain of +3.5/7.6 points over on-policy RLVR. Moreover, ExGRPO
stabilizes training on both stronger and weaker models where on-policy methods
fail. These results highlight principled experience management as a key
ingredient for efficient and scalable RLVR.
ExGRPO, a framework that prioritizes valuable reasoning experiences, improves and stabilizes reinforcement learning from verifiable rewards for large language models.
reinforcement learning from verifiable rewards RLVR on-policy training rollout experiences experience characteristics rollout correctness entropy ExGRPO Experiential Group Relative Policy Optimization mixed-policy objective exploration experience exploitation reasoning performance mathematical benchmarks general benchmarks