Critique-GRPO: Advancing LLM Reasoning with Natural Language and
  Numerical Feedback
Recent advances in reinforcement learning (RL) with numerical feedback, such
as scalar rewards, have significantly enhanced the complex reasoning
capabilities of large language models (LLMs). Despite this success, we identify
three key challenges encountered by RL with solely numerical feedback:
performance plateaus, limited effectiveness of self-reflection, and persistent
failures. We then demonstrate that RL-finetuned models, even after exhibiting
performance plateaus, can generate correct refinements on persistently failed
problems by leveraging natural language feedback in the form of critiques.
Building on this insight, we propose Critique-GRPO, an online RL framework that
integrates both natural language and numerical feedback for effective policy
optimization. Critique-GRPO enables LLMs to learn from initial responses and
critique-guided refinements simultaneously while maintaining exploration.
Extensive experiments using Qwen2.5-7B-Base and Qwen3-8B-Base show that
Critique-GRPO consistently outperforms supervised learning-based and RL-based
fine-tuning approaches across eight challenging mathematical, STEM, and general
reasoning tasks, improving average pass@1 scores by approximately 4.5% and 5%,
respectively. Notably, Critique-GRPO surpasses a strong baseline that
incorporates expert demonstrations within online RL. Further analysis reveals
two critical insights about policy exploration: (1) higher entropy does not
always guarantee efficient learning from exploration, and (2) longer responses
do not necessarily lead to more effective exploration.
684104d9ee7646c073776b2e
Xiaoying Zhang
684104d9ee7646c073776b2f
Hao Sun
684104d9ee7646c073776b30
Yipeng Zhang
extracted_pending
2025-06-05T02:45:46.851Z
666e91b1623133f1ce35acc5
/avatars/cc78520e6cfb83817c1d0c1ac867ebdd.svg
YipengZhang
user
YipengZhang
684104d9ee7646c073776b31
Kaituo Feng
684104d9ee7646c073776b32
Chaochao Lu
684104d9ee7646c073776b33
Chao Yang
684104d9ee7646c073776b34
Helen Meng
Critique-GRPO: Advancing LLM Reasoning with Natural Language and
  Numerical Feedback
Recent advances in reinforcement learning (RL) with numerical feedback, such
as scalar rewards, have significantly enhanced the complex reasoning
capabilities of large language models (LLMs). Despite this success, we identify
three key challenges encountered by RL with solely numerical feedback:
performance plateaus, limited effectiveness of self-reflection, and persistent
failures. We then demonstrate that RL-finetuned models, even after exhibiting
performance plateaus, can generate correct refinements on persistently failed
problems by leveraging natural language feedback in the form of critiques.
Building on this insight, we propose Critique-GRPO, an online RL framework that
integrates both natural language and numerical feedback for effective policy
optimization. Critique-GRPO enables LLMs to learn from initial responses and
critique-guided refinements simultaneously while maintaining exploration.
Extensive experiments using Qwen2.5-7B-Base and Qwen3-8B-Base show that
Critique-GRPO consistently outperforms supervised learning-based and RL-based
fine-tuning approaches across eight challenging mathematical, STEM, and general
reasoning tasks, improving average pass@1 scores by approximately 4.5% and 5%,
respectively. Notably, Critique-GRPO surpasses a strong baseline that
incorporates expert demonstrations within online RL. Further analysis reveals
two critical insights about policy exploration: (1) higher entropy does not
always guarantee efficient learning from exploration, and (2) longer responses
do not necessarily lead to more effective exploration.
Critique-GRPO, an RL framework combining numerical and natural language feedback, enhances LLM reasoning across tasks and outperforms existing methods.
reinforcement learning RL large language models LLMs scalar rewards performance plateaus self-reflection persistent failures natural language feedback critiques policy optimization Qwen2.5-7B-Base Qwen3-8B-Base pass@1 policy exploration entropy response length