RefCritic: Training Long Chain-of-Thought Critic Models with Refinement
  Feedback
With the rapid advancement of Large Language Models (LLMs), developing
effective critic modules for precise guidance has become crucial yet
challenging. In this paper, we initially demonstrate that supervised
fine-tuning for building critic modules (which is widely adopted in current
solutions) fails to genuinely enhance models' critique abilities, producing
superficial critiques with insufficient reflections and verifications. To
unlock the unprecedented critique capabilities, we propose RefCritic, a
long-chain-of-thought critic module based on reinforcement learning with dual
rule-based rewards: (1) instance-level correctness of solution judgments and
(2) refinement accuracies of the policy model based on critiques, aiming to
generate high-quality evaluations with actionable feedback that effectively
guides model refinement. We evaluate RefCritic on Qwen2.5-14B-Instruct and
DeepSeek-R1-Distill-Qwen-14B across five benchmarks. On critique and refinement
settings, RefCritic demonstrates consistent advantages across all benchmarks,
e.g., 6.8\% and 7.2\% gains on AIME25 for the respective base models. Notably,
under majority voting, policy models filtered by RefCritic show superior
scaling with increased voting numbers. Moreover, despite training on
solution-level supervision, RefCritic outperforms step-level supervised
approaches on ProcessBench, a benchmark to identify erroneous steps in
mathematical reasoning.
68805b3600f4b5a05f2fbe55
Qiaoyu Tang
admin_assigned
2025-07-23T13:34:14.864Z
64ed96f3067fbb625f83766a
/avatars/e8794ba19bdd965615bd9ead7df77d7d.svg
Qiaoyu Tang
user
TangQiaoYu
68805b3600f4b5a05f2fbe56
Hao Xiang
68805b3600f4b5a05f2fbe57
Le Yu
claimed_verified
2025-07-24T09:13:33.966Z
6455056fe4952d1c6cb45e49
/avatars/bbdcd4405497f11ff3774352277d1ec7.svg
Le Yu
user
vanillaOVO
68805b3600f4b5a05f2fbe58
Bowen Yu
68805b3600f4b5a05f2fbe59
Hongyu Lin
admin_assigned
2025-07-23T13:34:52.115Z
6711c702f858a456b4b9f3a4
/avatars/178e9567c3111ab22717c3c0dd003a6a.svg
Hongyu  Lin
user
sanmusunrise
68805b3600f4b5a05f2fbe5a
Yaojie Lu
admin_assigned
2025-07-23T13:34:46.021Z
6216496a9b34d2fb49144599
https://cdn-avatars.huggingface.co/v1/production/uploads/6216496a9b34d2fb49144599/41CKA_h1Ffj3RzVabSAkm.jpeg
Yaojie Lu
user
luyaojie
68805b3600f4b5a05f2fbe5b
Xianpei Han
admin_assigned
2025-07-23T13:34:38.403Z
65e99a77e71555ed193609cf
/avatars/38ceb127883944677665da967d17dd18.svg
Xianpei Han
user
xphan
68805b3600f4b5a05f2fbe5c
Le Sun
68805b3600f4b5a05f2fbe5d
Junyang Lin
admin_assigned
2025-07-23T13:34:21.167Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
RefCritic: Training Long Chain-of-Thought Critic Models with Refinement
  Feedback
With the rapid advancement of Large Language Models (LLMs), developing
effective critic modules for precise guidance has become crucial yet
challenging. In this paper, we initially demonstrate that supervised
fine-tuning for building critic modules (which is widely adopted in current
solutions) fails to genuinely enhance models' critique abilities, producing
superficial critiques with insufficient reflections and verifications. To
unlock the unprecedented critique capabilities, we propose RefCritic, a
long-chain-of-thought critic module based on reinforcement learning with dual
rule-based rewards: (1) instance-level correctness of solution judgments and
(2) refinement accuracies of the policy model based on critiques, aiming to
generate high-quality evaluations with actionable feedback that effectively
guides model refinement. We evaluate RefCritic on Qwen2.5-14B-Instruct and
DeepSeek-R1-Distill-Qwen-14B across five benchmarks. On critique and refinement
settings, RefCritic demonstrates consistent advantages across all benchmarks,
e.g., 6.8\% and 7.2\% gains on AIME25 for the respective base models. Notably,
under majority voting, policy models filtered by RefCritic show superior
scaling with increased voting numbers. Moreover, despite training on
solution-level supervision, RefCritic outperforms step-level supervised
approaches on ProcessBench, a benchmark to identify erroneous steps in
mathematical reasoning.
RefCritic, a reinforcement learning-based critic module with dual rule-based rewards, enhances model critique and refinement capabilities, demonstrating superior performance across multiple benchmarks compared to supervised fine-tuning methods.
Large Language Models critic modules supervised fine-tuning reinforcement learning rule-based rewards instance-level correctness refinement accuracies Qwen2.5-14B-Instruct DeepSeek-R1-Distill-Qwen-14B AIME25 ProcessBench majority voting