It Takes Two: Your GRPO Is Secretly DPO
Group Relative Policy Optimization (GRPO) is a prominent reinforcement
learning algorithm for post-training Large Language Models (LLMs). It is
commonly believed that GRPO necessitates a large group size to ensure stable
training via precise statistical estimation, which incurs substantial
computational overhead. In this work, we challenge this assumption by reframing
GRPO as a form of contrastive learning, which reveals a fundamental connection
to Direct Preference Optimization (DPO). Motivated by DPO's empirical success,
we investigate the minimal two-rollout case (2-GRPO), a configuration
previously deemed infeasible. We provide a rigorous theoretical analysis to
validate 2-GRPO and demonstrate empirically that it achieves performance on par
with 16-GRPO, despite using only 1/8 of the rollouts and reducing training time
by over 70%.
68de7e1b70ada21878c750a1
Yihong Wu
68de7e1b70ada21878c750a2
Liheng Ma
claimed_verified
2025-10-03T12:38:30.261Z
66e9b2b2244bcf9edc43dae5
/avatars/d2c47b5d0fabba49292d3b27fc2e5a5f.svg
Liheng Ma
user
lihengma
68de7e1b70ada21878c750a3
Lei Ding
68de7e1b70ada21878c750a4
Muzhi Li
68de7e1b70ada21878c750a5
Xinyu Wang
claimed_verified
2025-10-03T12:51:56.402Z
65e5fa6a55ed97ae47253e67
/avatars/363ea68811f5aab72c47637385128346.svg
Xinyu Wang
user
W4ng1204
68de7e1b70ada21878c750a6
Kejia Chen
68de7e1b70ada21878c750a7
Zhan Su
claimed_verified
2025-10-03T12:51:36.532Z
64dbf68ce1e953158d591adb
/avatars/30f48e9baaf2fbc7a9b0412a99b50bf2.svg
Zhan Su
user
zhan1993
68de7e1b70ada21878c750a8
Zhanguang Zhang
68de7e1b70ada21878c750a9
Chenyang Huang
68de7e1b70ada21878c750aa
Yingxue Zhang
68de7e1b70ada21878c750ab
Mark Coates
68de7e1b70ada21878c750ac
Jian-Yun Nie
It Takes Two: Your GRPO Is Secretly DPO
Group Relative Policy Optimization (GRPO) is a prominent reinforcement
learning algorithm for post-training Large Language Models (LLMs). It is
commonly believed that GRPO necessitates a large group size to ensure stable
training via precise statistical estimation, which incurs substantial
computational overhead. In this work, we challenge this assumption by reframing
GRPO as a form of contrastive learning, which reveals a fundamental connection
to Direct Preference Optimization (DPO). Motivated by DPO's empirical success,
we investigate the minimal two-rollout case (2-GRPO), a configuration
previously deemed infeasible. We provide a rigorous theoretical analysis to
validate 2-GRPO and demonstrate empirically that it achieves performance on par
with 16-GRPO, despite using only 1/8 of the rollouts and reducing training time
by over 70%.
Reframing Group Relative Policy Optimization as contrastive learning reveals its connection to Direct Preference Optimization, enabling minimal two-rollout GRPO to achieve performance comparable to larger group sizes with reduced computational cost.
reinforcement learning Large Language Models Group Relative Policy Optimization contrastive learning Direct Preference Optimization rollouts