GCPO: When Contrast Fails, Go Gold
Reinforcement learning has been widely applied to enhance the reasoning
capabilities of large language models. Extending the inference limits of
smaller models has become a prominent research focus. However, algorithms such
as Group Relative Policy Optimization (GRPO) suffer from a clear drawback: the
upper bound of a model's rollout responses is entirely determined by the model
itself, preventing the acquisition of knowledge from samples that are either
all incorrect or all correct. In this paper, we introduce Group Contrastive
Policy Optimization (GCPO), a method that incorporates external standard
reference answers. When the model cannot solve a problem, the reference answer
supplies the correct response, steering the model toward an unequivocally
accurate update direction. This approach offers two main advantages: (1) it
improves training efficiency by fully utilizing every sample; (2) it enables
the model to emulate the problem solving strategy of the reference answer
during training, thereby enhancing generalization in reasoning. GCPO achieves
outstanding results across multiple benchmark datasets, yielding substantial
improvements over the baseline model. Our code is available at:
https://github.com/AchoWu/GCPO.
68e8dadf95e8e6771df38c27
Hao Wu
claimed_verified
2025-10-10T10:08:28.678Z
64d0a05d2f1f9578a0405b9d
/avatars/30e69b345333e0bbe9149c8cf00c2fd9.svg
Hao Wu
user
Ach0
68e8dadf95e8e6771df38c28
Wei Liu
claimed_verified
2025-10-11T13:45:51.709Z
657181da907d85eaa5ba9f31
/avatars/601381d9f2b29e08193fc0439a92347d.svg
Wei Liu
user
Wilmido
GCPO: When Contrast Fails, Go Gold
Reinforcement learning has been widely applied to enhance the reasoning
capabilities of large language models. Extending the inference limits of
smaller models has become a prominent research focus. However, algorithms such
as Group Relative Policy Optimization (GRPO) suffer from a clear drawback: the
upper bound of a model's rollout responses is entirely determined by the model
itself, preventing the acquisition of knowledge from samples that are either
all incorrect or all correct. In this paper, we introduce Group Contrastive
Policy Optimization (GCPO), a method that incorporates external standard
reference answers. When the model cannot solve a problem, the reference answer
supplies the correct response, steering the model toward an unequivocally
accurate update direction. This approach offers two main advantages: (1) it
improves training efficiency by fully utilizing every sample; (2) it enables
the model to emulate the problem solving strategy of the reference answer
during training, thereby enhancing generalization in reasoning. GCPO achieves
outstanding results across multiple benchmark datasets, yielding substantial
improvements over the baseline model. Our code is available at:
https://github.com/AchoWu/GCPO.
Group Contrastive Policy Optimization (GCPO) enhances reinforcement learning for large language models by incorporating external reference answers, improving training efficiency and generalization.
Group Relative Policy Optimization (GRPO) Group Contrastive Policy Optimization (GCPO) reinforcement learning large language models rollout responses reference answers training efficiency generalization