ASPO: Asymmetric Importance Sampling Policy Optimization
Recent Large Language Model (LLM) post-training methods rely on token-level
clipping mechanisms during Reinforcement Learning (RL). However, we identify a
fundamental flaw in this Outcome-Supervised RL (OSRL) paradigm: the Importance
Sampling (IS) ratios of positive-advantage tokens are mismatched, leading to
unbalanced token weighting for positive and negative tokens. This mismatch
suppresses the update of low-probability tokens while over-amplifying already
high-probability ones. To address this, we propose Asymmetric Importance
Sampling Policy Optimization (ASPO), which uses a simple yet effective strategy
that flips the IS ratios of positive-advantage tokens, aligning their update
direction with the learning dynamics of negative ones. AIS further incorporates
a soft dual-clipping mechanism to stabilize extreme updates while maintaining
gradient flow. Comprehensive experiments on coding and mathematical reasoning
benchmarks demonstrate that ASPO significantly mitigates premature convergence,
improves training stability, and enhances final performance over strong
GRPO-based baselines. Our analysis provides new insights into the role of
token-level weighting in OSRL and highlights the critical importance of
correcting IS in LLM RL. The code and models of ASPO are available at
https://github.com/wizard-III/Archer2.0.
68e5e643975ac4c405ef213b
Jiakang Wang
68e5e643975ac4c405ef213c
Runze Liu
claimed_verified
2025-10-08T05:49:12.082Z
667187ba9ab144eb3ac43a1b
/avatars/db5558aa1c5160b9aee8b58573271959.svg
Runze Liu
user
RyanLiu112
68e5e643975ac4c405ef213d
Lei Lin
68e5e643975ac4c405ef213e
Wenping Hu
68e5e643975ac4c405ef213f
Xiu Li
68e5e643975ac4c405ef2140
Fuzheng Zhang
68e5e643975ac4c405ef2141
Guorui Zhou
68e5e643975ac4c405ef2142
Kun Gai
ASPO: Asymmetric Importance Sampling Policy Optimization
Recent Large Language Model (LLM) post-training methods rely on token-level
clipping mechanisms during Reinforcement Learning (RL). However, we identify a
fundamental flaw in this Outcome-Supervised RL (OSRL) paradigm: the Importance
Sampling (IS) ratios of positive-advantage tokens are mismatched, leading to
unbalanced token weighting for positive and negative tokens. This mismatch
suppresses the update of low-probability tokens while over-amplifying already
high-probability ones. To address this, we propose Asymmetric Importance
Sampling Policy Optimization (ASPO), which uses a simple yet effective strategy
that flips the IS ratios of positive-advantage tokens, aligning their update
direction with the learning dynamics of negative ones. AIS further incorporates
a soft dual-clipping mechanism to stabilize extreme updates while maintaining
gradient flow. Comprehensive experiments on coding and mathematical reasoning
benchmarks demonstrate that ASPO significantly mitigates premature convergence,
improves training stability, and enhances final performance over strong
GRPO-based baselines. Our analysis provides new insights into the role of
token-level weighting in OSRL and highlights the critical importance of
correcting IS in LLM RL. The code and models of ASPO are available at
https://github.com/wizard-III/Archer2.0.
ASPO addresses the imbalance in token weighting during OSRL by flipping Importance Sampling ratios and incorporating a soft dual-clipping mechanism, improving training stability and performance in LLMs.
Outcome-Supervised RL Importance Sampling Asymmetric Importance Sampling Policy Optimization soft dual-clipping mechanism token-level clipping low-probability tokens high-probability tokens premature convergence training stability GRPO-based baselines