Slow-Fast Policy Optimization: Reposition-Before-Update for LLM
  Reasoning
Reinforcement learning (RL) has become central to enhancing reasoning in
large language models (LLMs). Yet on-policy algorithms such as Group Relative
Policy Optimization (GRPO) often suffer in early training: noisy gradients from
low-quality rollouts lead to unstable updates and inefficient exploration. We
introduce Slow-Fast Policy Optimization (SFPO), a simple yet efficient
framework to address these limitations via decomposing each step into three
stages: a short fast trajectory of inner steps on the same batch, a reposition
mechanism to control off-policy drift, and a final slow correction. This
reposition-before-update design preserves the objective and rollout process
unchanged, making SFPO plug-compatible with existing policy-gradient pipelines.
Extensive experiments demonstrate that SFPO consistently improves stability,
reduces rollouts, and accelerates convergence of reasoning RL training.
Specifically, it outperforms GRPO by up to 2.80 points in average on math
reasoning benchmarks. It also achieves up to 4.93 fewer rollouts
and a 4.19 reduction in wall-clock time to match GRPO's best
accuracy.
68e49347e4e093a7044e4daf
Ziyan Wang
claimed_verified
2025-10-07T12:26:25.813Z
642c3805c885078517171815
/avatars/74d1179a28bd21beaa33d7ce437a5695.svg
Ziyan Wang
user
wzywp
68e49347e4e093a7044e4db0
Zheng Wang
claimed_verified
2025-10-08T08:01:57.124Z
656a90578dffbab5aff1be9d
/avatars/668b74323da098f6f9a7470aa422ec4b.svg
Wang
user
VictorZheng
68e49347e4e093a7044e4db1
Jie Fu
68e49347e4e093a7044e4db2
Xingwei Qu
68e49347e4e093a7044e4db3
Qi Cheng
68e49347e4e093a7044e4db4
Shengpu Tang
68e49347e4e093a7044e4db5
Minjia Zhang
68e49347e4e093a7044e4db6
Xiaoming Huo
Slow-Fast Policy Optimization: Reposition-Before-Update for LLM
  Reasoning
Reinforcement learning (RL) has become central to enhancing reasoning in
large language models (LLMs). Yet on-policy algorithms such as Group Relative
Policy Optimization (GRPO) often suffer in early training: noisy gradients from
low-quality rollouts lead to unstable updates and inefficient exploration. We
introduce Slow-Fast Policy Optimization (SFPO), a simple yet efficient
framework to address these limitations via decomposing each step into three
stages: a short fast trajectory of inner steps on the same batch, a reposition
mechanism to control off-policy drift, and a final slow correction. This
reposition-before-update design preserves the objective and rollout process
unchanged, making SFPO plug-compatible with existing policy-gradient pipelines.
Extensive experiments demonstrate that SFPO consistently improves stability,
reduces rollouts, and accelerates convergence of reasoning RL training.
Specifically, it outperforms GRPO by up to 2.80 points in average on math
reasoning benchmarks. It also achieves up to 4.93 fewer rollouts
and a 4.19 reduction in wall-clock time to match GRPO's best
accuracy.
Slow-Fast Policy Optimization (SFPO) enhances reinforcement learning training in large language models by improving stability, reducing rollouts, and accelerating convergence compared to Group Relative Policy Optimization (GRPO).
Reinforcement learning large language models Group Relative Policy Optimization Slow-Fast Policy Optimization policy-gradient pipelines on-policy algorithms noisy gradients low-quality rollouts unstable updates inefficient exploration reposition mechanism off-policy drift math reasoning benchmarks