Single-stream Policy Optimization
We revisit policy-gradient optimization for Large Language Models (LLMs) from
a single-stream perspective. Prevailing group-based methods like GRPO reduce
variance with on-the-fly baselines but suffer from critical flaws: frequent
degenerate groups erase learning signals, and synchronization barriers hinder
scalability. We introduce Single-stream Policy Optimization (SPO), which
eliminates these issues by design. SPO replaces per-group baselines with a
persistent, KL-adaptive value tracker and normalizes advantages globally across
the batch, providing a stable, low-variance learning signal for every sample.
Being group-free, SPO enables higher throughput and scales effectively in
long-horizon or tool-integrated settings where generation times vary.
Furthermore, the persistent value tracker naturally enables an adaptive
curriculum via prioritized sampling. Experiments using Qwen3-8B show that SPO
converges more smoothly and attains higher accuracy than GRPO, while
eliminating computation wasted on degenerate groups. Ablation studies confirm
that SPO's gains stem from its principled approach to baseline estimation and
advantage normalization, offering a more robust and efficient path for LLM
reasoning. Across five hard math benchmarks with Qwen3 8B, SPO improves the
average maj@32 by +3.4 percentage points (pp) over GRPO, driven by substantial
absolute point gains on challenging datasets, including +7.3 pp on BRUMO 25,
+4.4 pp on AIME 25, +3.3 pp on HMMT 25, and achieves consistent relative gain
in pass@k across the evaluated k values. SPO's success challenges the
prevailing trend of adding incidental complexity to RL algorithms, highlighting
a path where fundamental principles, not architectural workarounds, drive the
next wave of progress in LLM reasoning.
68ca17a96e0073c09bd1de5f
Zhongwen Xu
claimed_verified
2025-09-21T13:13:19.734Z
672db76fa34d64e774fc42c9
/avatars/be529f4bfcc40514697facbe8e874735.svg
Zhongwen Xu
user
zhongwenxu
68ca17a96e0073c09bd1de60
Zihan Ding
Single-stream Policy Optimization
We revisit policy-gradient optimization for Large Language Models (LLMs) from
a single-stream perspective. Prevailing group-based methods like GRPO reduce
variance with on-the-fly baselines but suffer from critical flaws: frequent
degenerate groups erase learning signals, and synchronization barriers hinder
scalability. We introduce Single-stream Policy Optimization (SPO), which
eliminates these issues by design. SPO replaces per-group baselines with a
persistent, KL-adaptive value tracker and normalizes advantages globally across
the batch, providing a stable, low-variance learning signal for every sample.
Being group-free, SPO enables higher throughput and scales effectively in
long-horizon or tool-integrated settings where generation times vary.
Furthermore, the persistent value tracker naturally enables an adaptive
curriculum via prioritized sampling. Experiments using Qwen3-8B show that SPO
converges more smoothly and attains higher accuracy than GRPO, while
eliminating computation wasted on degenerate groups. Ablation studies confirm
that SPO's gains stem from its principled approach to baseline estimation and
advantage normalization, offering a more robust and efficient path for LLM
reasoning. Across five hard math benchmarks with Qwen3 8B, SPO improves the
average maj@32 by +3.4 percentage points (pp) over GRPO, driven by substantial
absolute point gains on challenging datasets, including +7.3 pp on BRUMO 25,
+4.4 pp on AIME 25, +3.3 pp on HMMT 25, and achieves consistent relative gain
in pass@k across the evaluated k values. SPO's success challenges the
prevailing trend of adding incidental complexity to RL algorithms, highlighting
a path where fundamental principles, not architectural workarounds, drive the
next wave of progress in LLM reasoning.
Single-stream Policy Optimization (SPO) improves policy-gradient training for Large Language Models by eliminating group-based issues and providing a stable, low-variance learning signal, leading to better performance and efficiency.
policy-gradient optimization Large Language Models (LLMs) GRPO variance reduction on-the-fly baselines degenerate groups synchronization barriers Single-stream Policy Optimization (SPO) KL-adaptive value tracker advantage normalization adaptive curriculum prioritized sampling Qwen3-8B hard math benchmarks maj@32 BRUMO 25 AIME 25 HMMT 25 pass@$k$