Critique-Coder: Enhancing Coder Models by Critique Reinforcement
  Learning
Reinforcement Learning (RL) has emerged as a popular training paradigm,
particularly when paired with reasoning models. While effective, it primarily
focuses on generating responses and lacks mechanisms to explicitly foster
critique or reflection. Several recent studies, like Critique-Fine-Tuning (CFT)
and Critique-Guided-Distillation (CGD) have shown the benefits of explicitly
teaching LLMs how to critique. Motivated by them, we propose Critique
Reinforcement Learning (CRL), where the model is tasked with generating a
critique for a given (question, solution) pair. The reward is determined solely
by whether the final judgment label c in {True, False}
of the generated critique aligns with the ground-truth judgment c^*. Building
on this point, we introduce Critique-Coder, which is trained on a
hybrid of RL and CRL by substituting 20\% of the standard RL data with CRL
data. We fine-tune multiple models (Critique-Coder) and evaluate them
on different benchmarks to show their advantages over RL-only models. We show
that Critique-Coder consistently outperforms RL-only baselines on all
the evaluated benchmarks. Notably, our Critique-Coder-8B can reach
over 60\% on LiveCodeBench (v5), outperforming other reasoning models like
DeepCoder-14B and GPT-o1. Beyond code generation, Critique-Coder also
demonstrates enhanced general reasoning abilities, as evidenced by its better
performance on logic reasoning tasks from the BBEH dataset. This indicates that
the application of CRL on coding datasets enhances general reasoning and
critique abilities, which are transferable across a broad range of tasks.
Hence, we believe that CRL works as a great complement to standard RL for LLM
reasoning.
68db550cd2bf1f4b15ec74fe
Chi Ruan
68db550cd2bf1f4b15ec74ff
Dongfu Jiang
claimed_verified
2025-10-08T08:03:43.433Z
62567c86d444a9b5a0ec51c1
https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
Dongfu Jiang
user
DongfuJiang
68db550cd2bf1f4b15ec7500
Yubo Wang
68db550cd2bf1f4b15ec7501
Wenhu Chen
Critique-Coder: Enhancing Coder Models by Critique Reinforcement
  Learning
Reinforcement Learning (RL) has emerged as a popular training paradigm,
particularly when paired with reasoning models. While effective, it primarily
focuses on generating responses and lacks mechanisms to explicitly foster
critique or reflection. Several recent studies, like Critique-Fine-Tuning (CFT)
and Critique-Guided-Distillation (CGD) have shown the benefits of explicitly
teaching LLMs how to critique. Motivated by them, we propose Critique
Reinforcement Learning (CRL), where the model is tasked with generating a
critique for a given (question, solution) pair. The reward is determined solely
by whether the final judgment label c in {True, False}
of the generated critique aligns with the ground-truth judgment c^*. Building
on this point, we introduce Critique-Coder, which is trained on a
hybrid of RL and CRL by substituting 20\% of the standard RL data with CRL
data. We fine-tune multiple models (Critique-Coder) and evaluate them
on different benchmarks to show their advantages over RL-only models. We show
that Critique-Coder consistently outperforms RL-only baselines on all
the evaluated benchmarks. Notably, our Critique-Coder-8B can reach
over 60\% on LiveCodeBench (v5), outperforming other reasoning models like
DeepCoder-14B and GPT-o1. Beyond code generation, Critique-Coder also
demonstrates enhanced general reasoning abilities, as evidenced by its better
performance on logic reasoning tasks from the BBEH dataset. This indicates that
the application of CRL on coding datasets enhances general reasoning and
critique abilities, which are transferable across a broad range of tasks.
Hence, we believe that CRL works as a great complement to standard RL for LLM
reasoning.
Critique Reinforcement Learning (CRL) enhances LLMs by teaching them to generate critiques, leading to improved performance on code generation and logic reasoning tasks compared to standard RL.
Reinforcement Learning (RL) Critique-Fine-Tuning (CFT) Critique-Guided-Distillation (CGD) Critique Reinforcement Learning (CRL) Critique-Coder LiveCodeBench DeepCoder-14B GPT-o1 BBEH dataset