MAPO: Mixed Advantage Policy Optimization
Recent advances in reinforcement learning for foundation models, such as
Group Relative Policy Optimization (GRPO), have significantly improved the
performance of foundation models on reasoning tasks. Notably, the advantage
function serves as a central mechanism in GRPO for ranking the trajectory
importance. However, existing explorations encounter both advantage reversion
and advantage mirror problems, which hinder the reasonable advantage allocation
across different query samples. In this work, we propose an easy but effective
GRPO strategy, Mixed Advantage Policy Optimization (MAPO). We reveal that the
trajectory appears with different certainty and propose the advantage percent
deviation for samples with high-certainty trajectories. Furthermore, we
dynamically reweight the advantage function for samples with varying trajectory
certainty, thereby adaptively configuring the advantage function to account for
sample-specific characteristics. Comparison with related state-of-the-art
methods, along with ablation studies on different advantage variants, validates
the effectiveness of our approach.
68d363d20e215259d193b257
Wenke Huang
claimed_verified
2025-09-24T13:53:06.262Z
66b1dd6b93121096ffcfdab1
https://cdn-avatars.huggingface.co/v1/production/uploads/66b1dd6b93121096ffcfdab1/sOrbqdZhbp3ZP_cUGjhlN.jpeg
Wenke Huang
user
WilliamHuang91
68d363d20e215259d193b258
Quan Zhang
68d363d20e215259d193b259
Yiyang Fang
68d363d20e215259d193b25a
Jian Liang
68d363d20e215259d193b25b
Xuankun Rong
68d363d20e215259d193b25c
Huanjin Yao
68d363d20e215259d193b25d
Guancheng Wan
68d363d20e215259d193b25e
Ke Liang
68d363d20e215259d193b25f
Wenwen He
68d363d20e215259d193b260
Mingjun Li
68d363d20e215259d193b261
Leszek Rutkowski
68d363d20e215259d193b262
Mang Ye
68d363d20e215259d193b263
Bo Du
68d363d20e215259d193b264
Dacheng Tao
MAPO: Mixed Advantage Policy Optimization
Recent advances in reinforcement learning for foundation models, such as
Group Relative Policy Optimization (GRPO), have significantly improved the
performance of foundation models on reasoning tasks. Notably, the advantage
function serves as a central mechanism in GRPO for ranking the trajectory
importance. However, existing explorations encounter both advantage reversion
and advantage mirror problems, which hinder the reasonable advantage allocation
across different query samples. In this work, we propose an easy but effective
GRPO strategy, Mixed Advantage Policy Optimization (MAPO). We reveal that the
trajectory appears with different certainty and propose the advantage percent
deviation for samples with high-certainty trajectories. Furthermore, we
dynamically reweight the advantage function for samples with varying trajectory
certainty, thereby adaptively configuring the advantage function to account for
sample-specific characteristics. Comparison with related state-of-the-art
methods, along with ablation studies on different advantage variants, validates
the effectiveness of our approach.
Mixed Advantage Policy Optimization (MAPO) dynamically reweights the advantage function to improve trajectory ranking in reinforcement learning for foundation models.
Group Relative Policy Optimization (GRPO) advantage function trajectory importance advantage reversion advantage mirror problems Mixed Advantage Policy Optimization (MAPO) trajectory certainty advantage percent deviation sample-specific characteristics