GRPO-MA: Multi-Answer Generation in GRPO for Stable and Efficient
  Chain-of-Thought Training
Recent progress, such as DeepSeek-R1, has shown that the GRPO algorithm, a
Reinforcement Learning (RL) approach, can effectively train Chain-of-Thought
(CoT) reasoning in Large Language Models (LLMs) and Vision-Language Models
(VLMs). In this paper, we analyze three challenges of GRPO: gradient coupling
between thoughts and answers, sparse reward signals caused by limited parallel
sampling, and unstable advantage estimation. To mitigate these challenges, we
propose GRPO-MA, a simple yet theoretically grounded method that leverages
multi-answer generation from each thought process, enabling more robust and
efficient optimization. Theoretically, we show that the variance of thought
advantage decreases as the number of answers per thought increases.
Empirically, our gradient analysis confirms this effect, showing that GRPO-MA
reduces gradient spikes compared to GRPO. Experiments on math, code, and
diverse multimodal tasks demonstrate that GRPO-MA substantially improves
performance and training efficiency. Our ablation studies further reveal that
increasing the number of answers per thought consistently enhances model
performance.
68dbea034159d1f2418f97d6
Hongcheng Wang
68dbea034159d1f2418f97d7
Yinuo Huang
68dbea034159d1f2418f97d8
Sukai Wang
68dbea034159d1f2418f97d9
Guanghui Ren
claimed_verified
2025-10-01T10:24:02.718Z
646ec9b135f55eb49e405faa
/avatars/a17194be585d20e2a021e77a5a20e213.svg
Guanghui Ren
user
sundrops
68dbea034159d1f2418f97da
Hao Dong
GRPO-MA: Multi-Answer Generation in GRPO for Stable and Efficient
  Chain-of-Thought Training
Recent progress, such as DeepSeek-R1, has shown that the GRPO algorithm, a
Reinforcement Learning (RL) approach, can effectively train Chain-of-Thought
(CoT) reasoning in Large Language Models (LLMs) and Vision-Language Models
(VLMs). In this paper, we analyze three challenges of GRPO: gradient coupling
between thoughts and answers, sparse reward signals caused by limited parallel
sampling, and unstable advantage estimation. To mitigate these challenges, we
propose GRPO-MA, a simple yet theoretically grounded method that leverages
multi-answer generation from each thought process, enabling more robust and
efficient optimization. Theoretically, we show that the variance of thought
advantage decreases as the number of answers per thought increases.
Empirically, our gradient analysis confirms this effect, showing that GRPO-MA
reduces gradient spikes compared to GRPO. Experiments on math, code, and
diverse multimodal tasks demonstrate that GRPO-MA substantially improves
performance and training efficiency. Our ablation studies further reveal that
increasing the number of answers per thought consistently enhances model
performance.
GRPO-MA improves the training of Chain-of-Thought reasoning in LLMs and VLMs by addressing gradient coupling, sparse rewards, and unstable advantage estimation through multi-answer generation.
GRPO Reinforcement Learning Chain-of-Thought Large Language Models Vision-Language Models gradient coupling sparse reward signals unstable advantage estimation GRPO-MA multi-answer generation thought advantage gradient spikes