Geometric-Mean Policy Optimization
Recent advancements, such as Group Relative Policy Optimization (GRPO), have
enhanced the reasoning capabilities of large language models by optimizing the
arithmetic mean of token-level rewards. However, GRPO suffers from unstable
policy updates when processing tokens with outlier importance-weighted rewards,
which manifests as extreme importance sampling ratios during training, i.e.,
the ratio between the sampling probabilities assigned to a token by the current
and old policies. In this work, we propose Geometric-Mean Policy Optimization
(GMPO), a stabilized variant of GRPO. Instead of optimizing the arithmetic
mean, GMPO maximizes the geometric mean of token-level rewards, which is
inherently less sensitive to outliers and maintains a more stable range of
importance sampling ratio. In addition, we provide comprehensive theoretical
and experimental analysis to justify the design and stability benefits of GMPO.
Beyond improved stability, GMPO-7B outperforms GRPO by an average of 4.1% on
multiple mathematical benchmarks and 1.4% on multimodal reasoning benchmark,
including AIME24, AMC, MATH500, OlympiadBench, Minerva, and Geometry3K. Code is
available at https://github.com/callsys/GMPO.
68883a8eaf872d625c10c631
Yuzhong Zhao
68883a8eaf872d625c10c632
Yue Liu
68883a8eaf872d625c10c633
Junpeng Liu
claimed_verified
2025-07-29T07:08:16.611Z
6512a2e284bedffb5ac5a511
/avatars/e798f2c8633d77aea41fce684d749390.svg
Junpeng Liu
user
jeepliu
68883a8eaf872d625c10c634
Jingye Chen
68883a8eaf872d625c10c635
Xun Wu
68883a8eaf872d625c10c636
Yaru Hao
68883a8eaf872d625c10c637
Tengchao Lv
68883a8eaf872d625c10c638
Shaohan Huang
claimed_verified
2025-08-28T08:57:04.635Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
68883a8eaf872d625c10c639
Lei Cui
68883a8eaf872d625c10c63a
Qixiang Ye
68883a8eaf872d625c10c63b
Fang Wan
68883a8eaf872d625c10c63c
Furu Wei
Geometric-Mean Policy Optimization
Recent advancements, such as Group Relative Policy Optimization (GRPO), have
enhanced the reasoning capabilities of large language models by optimizing the
arithmetic mean of token-level rewards. However, GRPO suffers from unstable
policy updates when processing tokens with outlier importance-weighted rewards,
which manifests as extreme importance sampling ratios during training, i.e.,
the ratio between the sampling probabilities assigned to a token by the current
and old policies. In this work, we propose Geometric-Mean Policy Optimization
(GMPO), a stabilized variant of GRPO. Instead of optimizing the arithmetic
mean, GMPO maximizes the geometric mean of token-level rewards, which is
inherently less sensitive to outliers and maintains a more stable range of
importance sampling ratio. In addition, we provide comprehensive theoretical
and experimental analysis to justify the design and stability benefits of GMPO.
Beyond improved stability, GMPO-7B outperforms GRPO by an average of 4.1% on
multiple mathematical benchmarks and 1.4% on multimodal reasoning benchmark,
including AIME24, AMC, MATH500, OlympiadBench, Minerva, and Geometry3K. Code is
available at https://github.com/callsys/GMPO.
Geometric-Mean Policy Optimization (GMPO) stabilizes policy updates in large language models by maximizing the geometric mean of token-level rewards, outperforming Group Relative Policy Optimization (GRPO) on mathematical and multimodal reasoning benchmarks.
Group Relative Policy Optimization Geometric-Mean Policy Optimization token-level rewards importance-weighted rewards importance sampling ratios GMPO GRPO AIME24 AMC MATH500 OlympiadBench Minerva Geometry3K