Teaching Language Models to Critique via Reinforcement Learning
Teaching large language models (LLMs) to critique and refine their outputs is
crucial for building systems that can iteratively improve, yet it is
fundamentally limited by the ability to provide accurate judgments and
actionable suggestions. In this work, we study LLM critics for code generation
and propose CTRL, a framework for Critic
Training via Reinforcement Learning, which
trains a critic model to generate feedback that maximizes correction
performance for a fixed generator model without human supervision. Our results
demonstrate that critics trained with CTRL significantly enhance
pass rates and mitigate compounding errors across both base and stronger
generator models. Furthermore, we show that these critic models act as accurate
generative reward models and enable test-time scaling through iterative
critique-revision, achieving up to 106.1% relative improvements across
challenging code generation benchmarks.
67a5a8e595df68b0a167c298
Zhihui Xie
claimed_verified
2025-02-12T09:17:02.682Z
622f103fc78da4c7ebd7c887
/avatars/b0c7cd29835d92c2cd584947fcd5d520.svg
Xie
user
Zhihui
67a5a8e595df68b0a167c299
Jie chen
67a5a8e595df68b0a167c29a
Liyu Chen
67a5a8e595df68b0a167c29b
Weichao Mao
67a5a8e595df68b0a167c29c
Jingjing Xu
67a5a8e595df68b0a167c29d
Lingpeng Kong
Teaching Language Models to Critique via Reinforcement Learning
Teaching large language models (LLMs) to critique and refine their outputs is
crucial for building systems that can iteratively improve, yet it is
fundamentally limited by the ability to provide accurate judgments and
actionable suggestions. In this work, we study LLM critics for code generation
and propose CTRL, a framework for Critic
Training via Reinforcement Learning, which
trains a critic model to generate feedback that maximizes correction
performance for a fixed generator model without human supervision. Our results
demonstrate that critics trained with CTRL significantly enhance
pass rates and mitigate compounding errors across both base and stronger
generator models. Furthermore, we show that these critic models act as accurate
generative reward models and enable test-time scaling through iterative
critique-revision, achieving up to 106.1% relative improvements across
challenging code generation benchmarks.
A reinforcement learning framework trains a critic model to generate feedback for code generation, enhancing performance and mitigating errors.
LLMs code generation critic model reinforcement learning correction performance relative improvements generative reward models iterative critique-revision