Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of
  LLMs
Mathematical reasoning presents a significant challenge for Large Language
Models (LLMs) due to the extensive and precise chain of reasoning required for
accuracy. Ensuring the correctness of each reasoning step is critical. To
address this, we aim to enhance the robustness and factuality of LLMs by
learning from human feedback. However, Direct Preference Optimization (DPO) has
shown limited benefits for long-chain mathematical reasoning, as models
employing DPO struggle to identify detailed errors in incorrect answers. This
limitation stems from a lack of fine-grained process supervision. We propose a
simple, effective, and data-efficient method called Step-DPO, which treats
individual reasoning steps as units for preference optimization rather than
evaluating answers holistically. Additionally, we have developed a data
construction pipeline for Step-DPO, enabling the creation of a high-quality
dataset containing 10K step-wise preference pairs. We also observe that in DPO,
self-generated data is more effective than data generated by humans or GPT-4,
due to the latter's out-of-distribution nature. Our findings demonstrate that
as few as 10K preference data pairs and fewer than 500 Step-DPO training steps
can yield a nearly 3% gain in accuracy on MATH for models with over 70B
parameters. Notably, Step-DPO, when applied to Qwen2-72B-Instruct, achieves
scores of 70.8% and 94.0% on the test sets of MATH and GSM8K, respectively,
surpassing a series of closed-source models, including GPT-4-1106,
Claude-3-Opus, and Gemini-1.5-Pro. Our code, data, and models are available at
https://github.com/dvlab-research/Step-DPO.
667e1ec7b3072cb50c79ef59
Xin Lai
admin_assigned
2024-06-28T09:53:21.974Z
642bddc1fc41757877f68327
/avatars/f275237f36a112624d59a7e3f73237d3.svg
Xin Lai
user
xinlai
667e1ec7b3072cb50c79ef5a
Zhuotao Tian
admin_assigned
2024-06-28T09:53:16.366Z
64d104a37a7305c5895bd720
/avatars/2d9eff3a2dff6d02e45ae8964fb91f27.svg
zt tian
user
tianzhuotao
667e1ec7b3072cb50c79ef5b
Yukang Chen
admin_assigned
2024-06-28T09:53:04.262Z
65b31d77f32c79fea79ba4d1
/avatars/8c51c1ff337ac372c7caafa4fa5019dc.svg
Yukang Chen
user
Yukang2
667e1ec7b3072cb50c79ef5c
Senqiao Yang
admin_assigned
2024-06-28T09:52:57.935Z
6527b7280ae663e384eb8499
https://cdn-avatars.huggingface.co/v1/production/uploads/6527b7280ae663e384eb8499/73yF3eu2cUx7jVZrhXnXx.jpeg
Senqiao Yang
user
Senqiao
667e1ec7b3072cb50c79ef5d
Xiangru Peng
admin_assigned
2024-06-28T09:52:52.509Z
667e3aea97e3c9e6086a861f
/avatars/22d66b872173ce0935c2f4e6ba1e001d.svg
Xiangru Peng
user
pengxr
667e1ec7b3072cb50c79ef5e
Jiaya Jia
admin_assigned
2024-06-28T09:52:46.492Z
651ff0416f1df5466366400d
/avatars/caba60f52a490fc3aece6d4ae036d2c3.svg
Jim
user
jiajiaya
Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of
  LLMs
Mathematical reasoning presents a significant challenge for Large Language
Models (LLMs) due to the extensive and precise chain of reasoning required for
accuracy. Ensuring the correctness of each reasoning step is critical. To
address this, we aim to enhance the robustness and factuality of LLMs by
learning from human feedback. However, Direct Preference Optimization (DPO) has
shown limited benefits for long-chain mathematical reasoning, as models
employing DPO struggle to identify detailed errors in incorrect answers. This
limitation stems from a lack of fine-grained process supervision. We propose a
simple, effective, and data-efficient method called Step-DPO, which treats
individual reasoning steps as units for preference optimization rather than
evaluating answers holistically. Additionally, we have developed a data
construction pipeline for Step-DPO, enabling the creation of a high-quality
dataset containing 10K step-wise preference pairs. We also observe that in DPO,
self-generated data is more effective than data generated by humans or GPT-4,
due to the latter's out-of-distribution nature. Our findings demonstrate that
as few as 10K preference data pairs and fewer than 500 Step-DPO training steps
can yield a nearly 3% gain in accuracy on MATH for models with over 70B
parameters. Notably, Step-DPO, when applied to Qwen2-72B-Instruct, achieves
scores of 70.8% and 94.0% on the test sets of MATH and GSM8K, respectively,
surpassing a series of closed-source models, including GPT-4-1106,
Claude-3-Opus, and Gemini-1.5-Pro. Our code, data, and models are available at
https://github.com/dvlab-research/Step-DPO.
A novel method called Step-DPO enhances LLMs' mathematical reasoning by optimizing individual reasoning steps, achieving significant accuracy improvements with minimal data and training.
Large Language Models (LLMs) Direct Preference Optimization (DPO) Step-DPO preference optimization MATH GSM8K self-generated data