DeepCritic: Deliberate Critique with Large Language Models
As Large Language Models (LLMs) are rapidly evolving, providing accurate
feedback and scalable oversight on their outputs becomes an urgent and critical
problem. Leveraging LLMs as critique models to achieve automated supervision is
a promising solution. In this work, we focus on studying and enhancing the math
critique ability of LLMs. Current LLM critics provide critiques that are too
shallow and superficial on each step, leading to low judgment accuracy and
struggling to offer sufficient feedback for the LLM generator to correct
mistakes. To tackle this issue, we propose a novel and effective two-stage
framework to develop LLM critics that are capable of deliberately critiquing on
each reasoning step of math solutions. In the first stage, we utilize
Qwen2.5-72B-Instruct to generate 4.5K long-form critiques as seed data for
supervised fine-tuning. Each seed critique consists of deliberate step-wise
critiques that includes multi-perspective verifications as well as in-depth
critiques of initial critiques for each reasoning step. Then, we perform
reinforcement learning on the fine-tuned model with either existing
human-labeled data from PRM800K or our automatically annotated data obtained
via Monte Carlo sampling-based correctness estimation, to further incentivize
its critique ability. Our developed critique model built on Qwen2.5-7B-Instruct
not only significantly outperforms existing LLM critics (including the
same-sized DeepSeek-R1-distill models and GPT-4o) on various error
identification benchmarks, but also more effectively helps the LLM generator
refine erroneous steps through more detailed feedback.
68142e4a551709da9244e8d1
Wenkai Yang
claimed_verified
2025-05-02T06:34:18.531Z
64b7df742f5a966b973e25f7
/avatars/e24e7769188d441317b3b7d10ef8fd60.svg
Wenkai Yang
user
Keven16
68142e4a551709da9244e8d2
Jingwen Chen
68142e4a551709da9244e8d3
Yankai Lin
admin_assigned
2025-05-05T07:44:11.626Z
657a651e1433ea7d44de6397
/avatars/ccfc76f94595a38ff4a80f77c911eabf.svg
Yankai Lin
user
lyk423
68142e4a551709da9244e8d4
Ji-Rong Wen
admin_assigned
2025-05-05T07:44:17.877Z
64b8c89052b7353d8c6a1013
/avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
Ji-Rong Wen
user
jrwen
DeepCritic: Deliberate Critique with Large Language Models
As Large Language Models (LLMs) are rapidly evolving, providing accurate
feedback and scalable oversight on their outputs becomes an urgent and critical
problem. Leveraging LLMs as critique models to achieve automated supervision is
a promising solution. In this work, we focus on studying and enhancing the math
critique ability of LLMs. Current LLM critics provide critiques that are too
shallow and superficial on each step, leading to low judgment accuracy and
struggling to offer sufficient feedback for the LLM generator to correct
mistakes. To tackle this issue, we propose a novel and effective two-stage
framework to develop LLM critics that are capable of deliberately critiquing on
each reasoning step of math solutions. In the first stage, we utilize
Qwen2.5-72B-Instruct to generate 4.5K long-form critiques as seed data for
supervised fine-tuning. Each seed critique consists of deliberate step-wise
critiques that includes multi-perspective verifications as well as in-depth
critiques of initial critiques for each reasoning step. Then, we perform
reinforcement learning on the fine-tuned model with either existing
human-labeled data from PRM800K or our automatically annotated data obtained
via Monte Carlo sampling-based correctness estimation, to further incentivize
its critique ability. Our developed critique model built on Qwen2.5-7B-Instruct
not only significantly outperforms existing LLM critics (including the
same-sized DeepSeek-R1-distill models and GPT-4o) on various error
identification benchmarks, but also more effectively helps the LLM generator
refine erroneous steps through more detailed feedback.
A novel two-stage framework using Qwen2.5-72B-Instruct enhances LLMs' math critique ability by generating detailed step-wise critiques and applying reinforcement learning, resulting in better error identification and refinement.
Large Language Models (LLMs) critique models Qwen2.5-72B-Instruct supervised fine-tuning Monte Carlo sampling reinforcement learning DeepSeek-R1-distill GPT-4o error identification benchmarks