Segment Policy Optimization: Effective Segment-Level Credit Assignment
  in RL for Large Language Models
Enhancing the reasoning capabilities of large language models effectively
using reinforcement learning (RL) remains a crucial challenge. Existing
approaches primarily adopt two contrasting advantage estimation granularities:
Token-level methods (e.g., PPO) aim to provide the fine-grained advantage
signals but suffer from inaccurate estimation due to difficulties in training
an accurate critic model. On the other extreme, trajectory-level methods (e.g.,
GRPO) solely rely on a coarse-grained advantage signal from the final reward,
leading to imprecise credit assignment. To address these limitations, we
propose Segment Policy Optimization (SPO), a novel RL framework that leverages
segment-level advantage estimation at an intermediate granularity, achieving a
better balance by offering more precise credit assignment than trajectory-level
methods and requiring fewer estimation points than token-level methods,
enabling accurate advantage estimation based on Monte Carlo (MC) without a
critic model. SPO features three components with novel strategies: (1) flexible
segment partition; (2) accurate segment advantage estimation; and (3) policy
optimization using segment advantages, including a novel probability-mask
strategy. We further instantiate SPO for two specific scenarios: (1) SPO-chain
for short chain-of-thought (CoT), featuring novel cutpoint-based partition and
chain-based advantage estimation, achieving 6-12 percentage point
improvements in accuracy over PPO and GRPO on GSM8K. (2) SPO-tree for long CoT,
featuring novel tree-based advantage estimation, which significantly reduces
the cost of MC estimation, achieving 7-11 percentage point improvements
over GRPO on MATH500 under 2K and 4K context evaluation. We make our code
publicly available at https://github.com/AIFrameResearch/SPO.
6841771545662bb7d33ec3c7
Yiran Guo
claimed_verified
2025-06-05T12:41:51.856Z
652be5ba703b3743c2675448
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AOZ0lE57OYGd7r1ujZTr0.jpeg
Yiran Guo
user
gyr66
6841771545662bb7d33ec3c8
Lijie Xu
6841771545662bb7d33ec3c9
Jie Liu
6841771545662bb7d33ec3ca
Dan Ye
6841771545662bb7d33ec3cb
Shuang Qiu
claimed_verified
2025-06-05T12:41:49.841Z
6841784d883860ad472631c5
/avatars/290e1b9785f79cbbaccffdef24354282.svg
Shuang Qiu
user
shq-ml
Segment Policy Optimization: Effective Segment-Level Credit Assignment
  in RL for Large Language Models
Enhancing the reasoning capabilities of large language models effectively
using reinforcement learning (RL) remains a crucial challenge. Existing
approaches primarily adopt two contrasting advantage estimation granularities:
Token-level methods (e.g., PPO) aim to provide the fine-grained advantage
signals but suffer from inaccurate estimation due to difficulties in training
an accurate critic model. On the other extreme, trajectory-level methods (e.g.,
GRPO) solely rely on a coarse-grained advantage signal from the final reward,
leading to imprecise credit assignment. To address these limitations, we
propose Segment Policy Optimization (SPO), a novel RL framework that leverages
segment-level advantage estimation at an intermediate granularity, achieving a
better balance by offering more precise credit assignment than trajectory-level
methods and requiring fewer estimation points than token-level methods,
enabling accurate advantage estimation based on Monte Carlo (MC) without a
critic model. SPO features three components with novel strategies: (1) flexible
segment partition; (2) accurate segment advantage estimation; and (3) policy
optimization using segment advantages, including a novel probability-mask
strategy. We further instantiate SPO for two specific scenarios: (1) SPO-chain
for short chain-of-thought (CoT), featuring novel cutpoint-based partition and
chain-based advantage estimation, achieving 6-12 percentage point
improvements in accuracy over PPO and GRPO on GSM8K. (2) SPO-tree for long CoT,
featuring novel tree-based advantage estimation, which significantly reduces
the cost of MC estimation, achieving 7-11 percentage point improvements
over GRPO on MATH500 under 2K and 4K context evaluation. We make our code
publicly available at https://github.com/AIFrameResearch/SPO.
The Segment Policy Optimization (SPO) framework improves large language model reasoning via reinforcement learning by offering intermediate granularity advantage estimation, balancing precision and computational efficiency.
Segment Policy Optimization SPO token-level methods PPO trajectory-level methods GRPO segment-level advantage estimation Monte Carlo (MC) estimation flexible segment partition segment advantage estimation probability-mask strategy short chain-of-thought (CoT) cutpoint-based partition tree-based advantage estimation GSM8K MATH500