SuperCorrect: Supervising and Correcting Language Models with
  Error-Driven Insights
Large language models (LLMs) like GPT-4, PaLM, and LLaMA have shown
significant improvements in various reasoning tasks. However, smaller models
such as Llama-3-8B and DeepSeekMath-Base still struggle with complex
mathematical reasoning because they fail to effectively identify and correct
reasoning errors. Recent reflection-based methods aim to address these issues
by enabling self-reflection and self-correction, but they still face challenges
in independently detecting errors in their reasoning steps. To overcome these
limitations, we propose SuperCorrect, a novel two-stage framework that uses a
large teacher model to supervise and correct both the reasoning and reflection
processes of a smaller student model. In the first stage, we extract
hierarchical high-level and detailed thought templates from the teacher model
to guide the student model in eliciting more fine-grained reasoning thoughts.
In the second stage, we introduce cross-model collaborative direct preference
optimization (DPO) to enhance the self-correction abilities of the student
model by following the teacher's correction traces during training. This
cross-model DPO approach teaches the student model to effectively locate and
resolve erroneous thoughts with error-driven insights from the teacher model,
breaking the bottleneck of its thoughts and acquiring new skills and knowledge
to tackle challenging problems. Extensive experiments consistently demonstrate
our superiority over previous methods. Notably, our SuperCorrect-7B model
significantly surpasses powerful DeepSeekMath-7B by 7.8%/5.3% and
Qwen2.5-Math-7B by 15.1%/6.3% on MATH/GSM8K benchmarks, achieving new SOTA
performance among all 7B models. Code:
https://github.com/YangLing0818/SuperCorrect-llm
670c74c39f3ee99ddfb15805
Ling Yang
admin_assigned
2024-10-14T12:35:39.751Z
64fde4e252e82dd432b74ce9
https://cdn-avatars.huggingface.co/v1/production/uploads/64fde4e252e82dd432b74ce9/-CQZbBP7FsPPyawYrsi4z.jpeg
Ling Yang
user
Lingaaaaaaa
670c74c39f3ee99ddfb15806
Zhaochen Yu
admin_assigned
2024-10-14T12:35:45.584Z
64a131a7660cce8b86bf288d
/avatars/6c1a2475645a1a6ae3f804fe6c35a226.svg
zhao chen yu
user
chenyu01
670c74c39f3ee99ddfb15807
Tianjun Zhang
admin_assigned
2024-10-14T12:35:51.068Z
6374cd6b6ea8da14f8fef8dc
https://cdn-avatars.huggingface.co/v1/production/uploads/6374cd6b6ea8da14f8fef8dc/l13bg0tKDjCnUw3I895QZ.png
Tianjun Zhang
user
tianjunz
670c74c39f3ee99ddfb15808
Minkai Xu
admin_assigned
2024-10-14T12:35:57.450Z
64c0e950aa57599de1c75dad
/avatars/374d53317cbccc30fae70e5152ca13e0.svg
Minkai Xu
user
mkxu
670c74c39f3ee99ddfb15809
Joseph E. Gonzalez
admin_assigned
2024-10-14T12:36:03.351Z
645d2e8401f4eaab2a0878ce
/avatars/1273c5fb607b4b622a746a42692fa632.svg
Joseph E. Gonzalez
user
ProfJoeyG
670c74c39f3ee99ddfb1580a
Bin Cui
670c74c39f3ee99ddfb1580b
Shuicheng Yan
SuperCorrect: Supervising and Correcting Language Models with
  Error-Driven Insights
Large language models (LLMs) like GPT-4, PaLM, and LLaMA have shown
significant improvements in various reasoning tasks. However, smaller models
such as Llama-3-8B and DeepSeekMath-Base still struggle with complex
mathematical reasoning because they fail to effectively identify and correct
reasoning errors. Recent reflection-based methods aim to address these issues
by enabling self-reflection and self-correction, but they still face challenges
in independently detecting errors in their reasoning steps. To overcome these
limitations, we propose SuperCorrect, a novel two-stage framework that uses a
large teacher model to supervise and correct both the reasoning and reflection
processes of a smaller student model. In the first stage, we extract
hierarchical high-level and detailed thought templates from the teacher model
to guide the student model in eliciting more fine-grained reasoning thoughts.
In the second stage, we introduce cross-model collaborative direct preference
optimization (DPO) to enhance the self-correction abilities of the student
model by following the teacher's correction traces during training. This
cross-model DPO approach teaches the student model to effectively locate and
resolve erroneous thoughts with error-driven insights from the teacher model,
breaking the bottleneck of its thoughts and acquiring new skills and knowledge
to tackle challenging problems. Extensive experiments consistently demonstrate
our superiority over previous methods. Notably, our SuperCorrect-7B model
significantly surpasses powerful DeepSeekMath-7B by 7.8%/5.3% and
Qwen2.5-Math-7B by 15.1%/6.3% on MATH/GSM8K benchmarks, achieving new SOTA
performance among all 7B models. Code:
https://github.com/YangLing0818/SuperCorrect-llm
SuperCorrect uses a large teacher model to guide and correct a smaller student model's reasoning and reflection processes, significantly improving its performance in complex mathematical tasks.
large language models (LLMs) LLaMA DeepSeekMath Llama-3-8B hierarchical high-level detailed thought templates cross-model collaborative direct preference optimization (DPO) self-correction erroneous thoughts MATH/GSM8K benchmarks