BroRL: Scaling Reinforcement Learning via Broadened Exploration
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key
ingredient for unlocking complex reasoning capabilities in large language
models. Recent work ProRL has shown promise in scaling RL by increasing the
number of training steps. However, performance plateaus after thousands of
steps, with clear diminishing returns from allocating more computation to
additional training. In this work, we investigate a complementary paradigm for
scaling RL, BroR-Lincreasing the number of rollouts per example to hundreds to
exhaustively Broaden exploration, which yields continuous performance gains
beyond the saturation point observed in ProRL when scaling the number of
training steps. Our approach is motivated by a mass balance equation analysis
allowing us to characterize the rate of change in probability mass for correct
and incorrect tokens during the reinforcement process. We show that under a
one-step RL assumption, sampled rollout tokens always contribute to
correct-mass expansion, while unsampled tokens outside rollouts may lead to
gains or losses depending on their distribution and the net reward balance.
Importantly, as the number of rollouts per example N increases, the effect of
unsampled terms diminishes, ensuring overall correct-mass expansion. To
validate our theoretical analysis, we conduct simulations under more relaxed
conditions and find that a sufficiently large rollout size N-corresponding to
ample exploration-guarantees an increase in the probability mass of all correct
tokens. Empirically, BroRL revives models saturated after 3K ProRL training
steps and demonstrates robust, continuous improvement, achieving
state-of-the-art results for the 1.5B model across diverse benchmarks.
68de14816024653e8a3ed195
Jian Hu
68de14816024653e8a3ed196
Mingjie Liu
68de14816024653e8a3ed197
Ximing Lu
68de14816024653e8a3ed198
Fang Wu
68de14816024653e8a3ed199
Zaid Harchaoui
68de14816024653e8a3ed19a
Shizhe Diao
68de14816024653e8a3ed19b
Yejin Choi
68de14816024653e8a3ed19c
Pavlo Molchanov
68de14816024653e8a3ed19d
Jun Yang
68de14816024653e8a3ed19e
Jan Kautz
68de14816024653e8a3ed19f
Yi Dong
BroRL: Scaling Reinforcement Learning via Broadened Exploration
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key
ingredient for unlocking complex reasoning capabilities in large language
models. Recent work ProRL has shown promise in scaling RL by increasing the
number of training steps. However, performance plateaus after thousands of
steps, with clear diminishing returns from allocating more computation to
additional training. In this work, we investigate a complementary paradigm for
scaling RL, BroR-Lincreasing the number of rollouts per example to hundreds to
exhaustively Broaden exploration, which yields continuous performance gains
beyond the saturation point observed in ProRL when scaling the number of
training steps. Our approach is motivated by a mass balance equation analysis
allowing us to characterize the rate of change in probability mass for correct
and incorrect tokens during the reinforcement process. We show that under a
one-step RL assumption, sampled rollout tokens always contribute to
correct-mass expansion, while unsampled tokens outside rollouts may lead to
gains or losses depending on their distribution and the net reward balance.
Importantly, as the number of rollouts per example N increases, the effect of
unsampled terms diminishes, ensuring overall correct-mass expansion. To
validate our theoretical analysis, we conduct simulations under more relaxed
conditions and find that a sufficiently large rollout size N-corresponding to
ample exploration-guarantees an increase in the probability mass of all correct
tokens. Empirically, BroRL revives models saturated after 3K ProRL training
steps and demonstrates robust, continuous improvement, achieving
state-of-the-art results for the 1.5B model across diverse benchmarks.
BroRL enhances reinforcement learning by increasing rollouts per example, overcoming performance plateaus and achieving state-of-the-art results in large language models.
Reinforcement Learning with Verifiable Rewards RLVR ProRL rollouts exploration mass balance equation probability mass correct tokens incorrect tokens one-step RL net reward balance correct-mass expansion unsampled terms