Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale
  Data on LLMs?
Reinforcement learning has been central to recent advances in large language
model reasoning, but most algorithms rely on on-policy training that demands
fresh rollouts at every update, limiting efficiency and scalability.
Asynchronous RL systems alleviate this by decoupling rollout generation from
training, yet their effectiveness hinges on tolerating large staleness in
rollout data, a setting where existing methods either degrade in performance or
collapse. We revisit this challenge and uncover a prosperity-before-collapse
phenomenon: stale data can be as informative as on-policy data if exploited
properly. Building on this insight, we introduce M2PO (Second-Moment Trust
Policy Optimization), which constrains the second moment of importance weights
to suppress only extreme outliers while preserving informative updates.
Notably, M2PO sharply reduces the fraction of clipped tokens under high
staleness (from 1.22% to 0.06% over training), precisely masking high-variance
tokens while maintaining stable optimization. Extensive evaluation across six
models (from 1.7B to 32B) and eight benchmarks shows that M2PO delivers stable
off-policy training even with data stale by at least 256 model updates and
matches on-policy performance.
68e5597d975ac4c405ef1f25
Haizhong Zheng
68e5597d975ac4c405ef1f26
Jiawei Zhao
68e5597d975ac4c405ef1f27
Bedi Chen
Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale
  Data on LLMs?
Reinforcement learning has been central to recent advances in large language
model reasoning, but most algorithms rely on on-policy training that demands
fresh rollouts at every update, limiting efficiency and scalability.
Asynchronous RL systems alleviate this by decoupling rollout generation from
training, yet their effectiveness hinges on tolerating large staleness in
rollout data, a setting where existing methods either degrade in performance or
collapse. We revisit this challenge and uncover a prosperity-before-collapse
phenomenon: stale data can be as informative as on-policy data if exploited
properly. Building on this insight, we introduce M2PO (Second-Moment Trust
Policy Optimization), which constrains the second moment of importance weights
to suppress only extreme outliers while preserving informative updates.
Notably, M2PO sharply reduces the fraction of clipped tokens under high
staleness (from 1.22% to 0.06% over training), precisely masking high-variance
tokens while maintaining stable optimization. Extensive evaluation across six
models (from 1.7B to 32B) and eight benchmarks shows that M2PO delivers stable
off-policy training even with data stale by at least 256 model updates and
matches on-policy performance.
M2PO, a reinforcement learning algorithm, enables stable off-policy training with stale data by constraining the second moment of importance weights, achieving performance comparable to on-policy methods.
reinforcement learning on-policy training asynchronous RL rollout generation staleness importance weights M2PO Second-Moment Trust Policy Optimization clipped tokens high-variance tokens off-policy training