BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via
  Balanced Policy Optimization with Adaptive Clipping
Reinforcement learning (RL) has recently become the core paradigm for
aligning and strengthening large language models (LLMs). Yet, applying RL in
off-policy settings--where stale data from past policies are used for
training--improves sample efficiency, but remains challenging: policy entropy
declines sharply, optimization often becomes unstable and may even collapse.
Through theoretical and empirical analysis, we identify two key insights: (i)
an imbalance in optimization, where negative-advantage samples dominate the
policy gradient, suppressing useful behaviors and risking gradient explosions;
and (ii) the derived Entropy-Clip Rule, which reveals that the fixed clipping
mechanism in PPO-like objectives systematically blocks entropy-increasing
updates, thereby driving the policy toward over-exploitation at the expense of
exploration. Building on these insights, we propose BAlanced Policy
Optimization with Adaptive Clipping (BAPO), a simple yet effective method that
dynamically adjusts clipping bounds to adaptively re-balance positive and
negative contributions, preserve entropy, and stabilize RL optimization. Across
diverse off-policy scenarios--including sample replay and partial rollout--BAPO
achieves fast, stable, and data-efficient training. On AIME 2024 and AIME 2025
benchmarks, our 7B BAPO model surpasses open-source counterparts such as
SkyWork-OR1-7B, while our 32B BAPO model not only achieves state-of-the-art
results among models of the same scale but also outperforms leading proprietary
systems like o3-mini and Gemini-2.5-Flash-Thinking.
68f9a187b9b2e4ae04673870
Zhiheng Xi
68f9a187b9b2e4ae04673871
Xin Guo
68f9a187b9b2e4ae04673872
Yang Nan
68f9a187b9b2e4ae04673873
Enyu Zhou
68f9a187b9b2e4ae04673874
Junrui Shen
claimed_verified
2025-10-23T14:47:23.670Z
68f9b5fa0313464f8c8132d1
/avatars/2bb01b4ef391140daa8299f6a55c19a3.svg
Shen
user
Vindicaters
68f9a187b9b2e4ae04673875
Wenxiang Chen
68f9a187b9b2e4ae04673876
Jiaqi Liu
68f9a187b9b2e4ae04673877
Jixuan Huang
68f9a187b9b2e4ae04673878
Zhihao Zhang
68f9a187b9b2e4ae04673879
Honglin Guo
68f9a187b9b2e4ae0467387a
Xun Deng
68f9a187b9b2e4ae0467387b
Zhikai Lei
68f9a187b9b2e4ae0467387c
Miao Zheng
68f9a187b9b2e4ae0467387d
Guoteng Wang
68f9a187b9b2e4ae0467387e
Shuo Zhang
claimed_verified
2025-10-24T16:13:02.711Z
6334f2f1259c518276efa730
https://cdn-avatars.huggingface.co/v1/production/uploads/6334f2f1259c518276efa730/z_SH_OBkDyj4RCN9mqsKS.jpeg
Shuo Zhang
user
Meteonis
68f9a187b9b2e4ae0467387f
Peng Sun
claimed_verified
2025-10-24T16:13:05.008Z
637ca33abf9c9748e3be2649
/avatars/5ce312d2343715f07324dabefea624ad.svg
Peng Sun
user
voidmain
68f9a187b9b2e4ae04673880
Rui Zheng
68f9a187b9b2e4ae04673881
Hang Yan
68f9a187b9b2e4ae04673882
Tao Gui
68f9a187b9b2e4ae04673883
Qi Zhang
68f9a187b9b2e4ae04673884
Xuanjing Huang
BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via
  Balanced Policy Optimization with Adaptive Clipping
Reinforcement learning (RL) has recently become the core paradigm for
aligning and strengthening large language models (LLMs). Yet, applying RL in
off-policy settings--where stale data from past policies are used for
training--improves sample efficiency, but remains challenging: policy entropy
declines sharply, optimization often becomes unstable and may even collapse.
Through theoretical and empirical analysis, we identify two key insights: (i)
an imbalance in optimization, where negative-advantage samples dominate the
policy gradient, suppressing useful behaviors and risking gradient explosions;
and (ii) the derived Entropy-Clip Rule, which reveals that the fixed clipping
mechanism in PPO-like objectives systematically blocks entropy-increasing
updates, thereby driving the policy toward over-exploitation at the expense of
exploration. Building on these insights, we propose BAlanced Policy
Optimization with Adaptive Clipping (BAPO), a simple yet effective method that
dynamically adjusts clipping bounds to adaptively re-balance positive and
negative contributions, preserve entropy, and stabilize RL optimization. Across
diverse off-policy scenarios--including sample replay and partial rollout--BAPO
achieves fast, stable, and data-efficient training. On AIME 2024 and AIME 2025
benchmarks, our 7B BAPO model surpasses open-source counterparts such as
SkyWork-OR1-7B, while our 32B BAPO model not only achieves state-of-the-art
results among models of the same scale but also outperforms leading proprietary
systems like o3-mini and Gemini-2.5-Flash-Thinking.
BAlanced Policy Optimization with Adaptive Clipping (BAPO) addresses challenges in off-policy reinforcement learning by dynamically adjusting clipping bounds to improve sample efficiency, stability, and performance in large language models.
reinforcement learning large language models off-policy settings policy entropy policy gradient Entropy-Clip Rule PPO-like objectives gradient explosions sample replay partial rollout AIME 2024 AIME 2025 SkyWork-OR1-7B o3-mini Gemini-2.5-Flash-Thinking