RL-PLUS: Countering Capability Boundary Collapse of LLMs in
  Reinforcement Learning with Hybrid-policy Optimization
Reinforcement Learning with Verifiable Reward (RLVR) has significantly
advanced the complex reasoning abilities of Large Language Models (LLMs).
However, it struggles to break through the inherent capability boundaries of
the base LLM, due to its essentially on-policy strategy coupled with LLM's
immense action space and sparse reward. Critically, RLVR can lead to the
capability boundary collapse, narrowing the LLM's problem-solving scope. To
address this problem, we propose RL-PLUS, a novel hybrid-policy optimization
approach for LLMs that synergizes internal exploitation with external data to
achieve stronger reasoning capabilities and surpass the boundaries of base
models. RL-PLUS integrates two core components, i.e., Multiple Importance
Sampling to address distributional mismatch from external data, and
Exploration-Based Advantage Function to guide the model towards high-value,
unexplored reasoning paths. We provide both theoretical analysis and extensive
experiments to demonstrate the superiority and generalizability of our
approach. Compared with existing RLVR methods, RL-PLUS achieves 1)
state-of-the-art performance on six math reasoning benchmarks; 2) superior
performance on six out-of-distribution reasoning tasks; 3) consistent and
significant gains across diverse model families, with average relative
improvements up to 69.2\%. Moreover, the analysis of Pass@k curves indicates
that RL-PLUS effectively resolves the capability boundary collapse problem.
68944a32741a16f544fbcfdc
Yihong Dong
68944a32741a16f544fbcfdd
Xue Jiang
68944a32741a16f544fbcfde
Yongding Tao
68944a32741a16f544fbcfdf
Huanyu Liu
68944a32741a16f544fbcfe0
Kechi Zhang
68944a32741a16f544fbcfe1
Lili Mou
68944a32741a16f544fbcfe2
Rongyu Cao
68944a32741a16f544fbcfe3
Yingwei Ma
68944a32741a16f544fbcfe4
Jue Chen
68944a32741a16f544fbcfe5
Binhua Li
68944a32741a16f544fbcfe6
Zhi Jin
68944a32741a16f544fbcfe7
Fei Huang
68944a32741a16f544fbcfe8
Yongbin Li
68944a32741a16f544fbcfe9
Ge Li
RL-PLUS: Countering Capability Boundary Collapse of LLMs in
  Reinforcement Learning with Hybrid-policy Optimization
Reinforcement Learning with Verifiable Reward (RLVR) has significantly
advanced the complex reasoning abilities of Large Language Models (LLMs).
However, it struggles to break through the inherent capability boundaries of
the base LLM, due to its essentially on-policy strategy coupled with LLM's
immense action space and sparse reward. Critically, RLVR can lead to the
capability boundary collapse, narrowing the LLM's problem-solving scope. To
address this problem, we propose RL-PLUS, a novel hybrid-policy optimization
approach for LLMs that synergizes internal exploitation with external data to
achieve stronger reasoning capabilities and surpass the boundaries of base
models. RL-PLUS integrates two core components, i.e., Multiple Importance
Sampling to address distributional mismatch from external data, and
Exploration-Based Advantage Function to guide the model towards high-value,
unexplored reasoning paths. We provide both theoretical analysis and extensive
experiments to demonstrate the superiority and generalizability of our
approach. Compared with existing RLVR methods, RL-PLUS achieves 1)
state-of-the-art performance on six math reasoning benchmarks; 2) superior
performance on six out-of-distribution reasoning tasks; 3) consistent and
significant gains across diverse model families, with average relative
improvements up to 69.2\%. Moreover, the analysis of Pass@k curves indicates
that RL-PLUS effectively resolves the capability boundary collapse problem.
RL-PLUS, a hybrid-policy optimization approach, enhances LLM reasoning capabilities by integrating Multiple Importance Sampling and Exploration-Based Advantage Function, outperforming RLVR on various benchmarks and resolving capability boundary collapse.
Reinforcement Learning with Verifiable Reward (RLVR) Large Language Models (LLMs) hybrid-policy optimization Multiple Importance Sampling Exploration-Based Advantage Function capability boundary collapse math reasoning benchmarks out-of-distribution reasoning tasks