Critique Fine-Tuning: Learning to Critique is More Effective than
  Learning to Imitate
Supervised Fine-Tuning (SFT) is commonly used to train language models to
imitate annotated responses for given instructions. In this paper, we challenge
this paradigm and propose Critique Fine-Tuning (CFT), a strategy where models
learn to critique noisy responses rather than simply imitate correct ones.
Inspired by human learning processes that emphasize critical thinking, CFT
encourages deeper analysis and nuanced understanding-traits often overlooked by
standard SFT. To validate the effectiveness of CFT, we construct a 50K-sample
dataset from WebInstruct, using GPT-4o as the teacher to generate critiques in
the form of (input=[query; noisy response], output=critique). CFT on this
dataset yields a consistent 4-10% improvement over SFT on six math benchmarks
with different base models like Qwen2.5, Qwen2.5-Math and DeepSeek-Math. We
further expand to MetaMath and NuminaMath datasets and observe similar gains
over SFT. Notably, our Qwen2.5-Math-CFT model-trained on just 50K
samples-matches or outperforms competitive models such as AceMath and
Qwen2.5-Math-Instruct on most benchmarks, both of which use over 2M samples.
Ablation studies show that CFT is robust to the source of noisy response and
teacher critique model. Through these findings, we argue that critique-based
training offers a more effective alternative to advance the reasoning of
language models.
679ae76cf211c66bd702f5d5
Yubo Wang
claimed_verified
2025-01-30T08:39:49.375Z
636a35eff8d9af4aea181608
https://cdn-avatars.huggingface.co/v1/production/uploads/636a35eff8d9af4aea181608/s9GFJYd_QXVbg0Lb4JpKj.jpeg
yubo
user
ubowang
679ae76cf211c66bd702f5d6
Xiang Yue
claimed_verified
2025-01-30T15:17:01.780Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
679ae76cf211c66bd702f5d7
Wenhu Chen
extracted_pending
2025-01-30T02:43:59.302Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
Critique Fine-Tuning: Learning to Critique is More Effective than
  Learning to Imitate
Supervised Fine-Tuning (SFT) is commonly used to train language models to
imitate annotated responses for given instructions. In this paper, we challenge
this paradigm and propose Critique Fine-Tuning (CFT), a strategy where models
learn to critique noisy responses rather than simply imitate correct ones.
Inspired by human learning processes that emphasize critical thinking, CFT
encourages deeper analysis and nuanced understanding-traits often overlooked by
standard SFT. To validate the effectiveness of CFT, we construct a 50K-sample
dataset from WebInstruct, using GPT-4o as the teacher to generate critiques in
the form of (input=[query; noisy response], output=critique). CFT on this
dataset yields a consistent 4-10% improvement over SFT on six math benchmarks
with different base models like Qwen2.5, Qwen2.5-Math and DeepSeek-Math. We
further expand to MetaMath and NuminaMath datasets and observe similar gains
over SFT. Notably, our Qwen2.5-Math-CFT model-trained on just 50K
samples-matches or outperforms competitive models such as AceMath and
Qwen2.5-Math-Instruct on most benchmarks, both of which use over 2M samples.
Ablation studies show that CFT is robust to the source of noisy response and
teacher critique model. Through these findings, we argue that critique-based
training offers a more effective alternative to advance the reasoning of
language models.
Critique Fine-Tuning (CFT) improves language model performance by training them to critique noisy responses, showing better reasoning ability than standard Supervised Fine-Tuning (SFT).
Critique Fine-Tuning CFT Supervised Fine-Tuning SFT WebInstruct GPT-4 Qwen2.5 Qwen2.5-Math DeepSeek-Math MetaMath NuminaMath AceMath Qwen2.5-Math-Instruct