CARFT: Boosting LLM Reasoning via Contrastive Learning with Annotated
  Chain-of-Thought-based Reinforced Fine-Tuning
Reasoning capability plays a significantly critical role in the the broad
applications of Large Language Models (LLMs). To enhance the reasoning
performance of LLMs, diverse Reinforcement Learning (RL)-based fine-tuning
approaches have been proposed to address the limited generalization capability
of LLMs trained solely via Supervised Fine-Tuning (SFT). Despite their
effectiveness, two major limitations hinder the advancement of LLMs. First,
vanilla RL-based approaches ignore annotated Chain-of-Thought (CoT) and
incorporate unstable reasoning path sampling, which typically results in model
collapse, unstable training process, and suboptimal performance. Second,
existing SFT approaches generally overemphasize the annotated CoT, potentially
leading to performance degradation due to insufficient exploitation of
potential CoT. In this paper, we propose a Contrastive learning with annotated
CoT-based Reinforced Fine-Tuning approach, i.e., , to enhance the
reasoning performance of LLMs while addressing the aforementioned limitations.
Specifically, we propose learning a representation for each CoT. Based on this
representation, we design novel contrastive signals to guide the fine-tuning
process. Our approach not only fully exploits the available annotated CoT but
also stabilizes the fine-tuning procedure by incorporating an additional
unsupervised learning signal. We conduct comprehensive experiments and in-depth
analysis with three baseline approaches, two foundation models, and two
datasets to demonstrate significant advantages of  in terms of
robustness, performance (up to 10.15\%), and efficiency (up to 30.62\%). Code
is available at https://github.com/WNQzhu/CARFT.
68ac43be86b21a0e2e358bec
Wenqiao Zhu
claimed_verified
2025-08-25T12:37:27.698Z
648add6aff6123185eb185a8
/avatars/e37dfa680c1bb86c721165f03eb79e97.svg
WNQzhu
user
Qlisp
68ac43be86b21a0e2e358bed
Ji Liu
68ac43be86b21a0e2e358bee
Rongjuncheng Zhang
68ac43be86b21a0e2e358bef
Haipang Wu
68ac43be86b21a0e2e358bf0
Yulun Zhang
CARFT: Boosting LLM Reasoning via Contrastive Learning with Annotated
  Chain-of-Thought-based Reinforced Fine-Tuning
Reasoning capability plays a significantly critical role in the the broad
applications of Large Language Models (LLMs). To enhance the reasoning
performance of LLMs, diverse Reinforcement Learning (RL)-based fine-tuning
approaches have been proposed to address the limited generalization capability
of LLMs trained solely via Supervised Fine-Tuning (SFT). Despite their
effectiveness, two major limitations hinder the advancement of LLMs. First,
vanilla RL-based approaches ignore annotated Chain-of-Thought (CoT) and
incorporate unstable reasoning path sampling, which typically results in model
collapse, unstable training process, and suboptimal performance. Second,
existing SFT approaches generally overemphasize the annotated CoT, potentially
leading to performance degradation due to insufficient exploitation of
potential CoT. In this paper, we propose a Contrastive learning with annotated
CoT-based Reinforced Fine-Tuning approach, i.e., , to enhance the
reasoning performance of LLMs while addressing the aforementioned limitations.
Specifically, we propose learning a representation for each CoT. Based on this
representation, we design novel contrastive signals to guide the fine-tuning
process. Our approach not only fully exploits the available annotated CoT but
also stabilizes the fine-tuning procedure by incorporating an additional
unsupervised learning signal. We conduct comprehensive experiments and in-depth
analysis with three baseline approaches, two foundation models, and two
datasets to demonstrate significant advantages of  in terms of
robustness, performance (up to 10.15\%), and efficiency (up to 30.62\%). Code
is available at https://github.com/WNQzhu/CARFT.
A contrastive learning approach with annotated Chain-of-Thought enhances the reasoning performance of Large Language Models through stable fine-tuning and robust contrastive signals.
Reinforcement Learning Supervised Fine-Tuning Chain-of-Thought model collapse contrastive learning contrastive signals unsupervised learning