Self-Training with Direct Preference Optimization Improves
  Chain-of-Thought Reasoning
Effective training of language models (LMs) for mathematical reasoning tasks
demands high-quality supervised fine-tuning data. Besides obtaining annotations
from human experts, a common alternative is sampling from larger and more
powerful LMs. However, this knowledge distillation approach can be costly and
unstable, particularly when relying on closed-source, proprietary LMs like
GPT-4, whose behaviors are often unpredictable. In this work, we demonstrate
that the reasoning abilities of small-scale LMs can be enhanced through
self-training, a process where models learn from their own outputs. We also
show that the conventional self-training can be further augmented by a
preference learning algorithm called Direct Preference Optimization (DPO). By
integrating DPO into self-training, we leverage preference data to guide LMs
towards more accurate and diverse chain-of-thought reasoning. We evaluate our
method across various mathematical reasoning tasks using different base models.
Our experiments show that this approach not only improves LMs' reasoning
performance but also offers a more cost-effective and scalable solution
compared to relying on large proprietary LMs.
66a740d7577ca69aa4c1c4a8
Tianduo Wang
admin_assigned
2024-07-30T08:38:18.265Z
6352aa7b6cfb8f149814de5e
https://cdn-avatars.huggingface.co/v1/production/uploads/1666361939036-noauth.jpeg
Tianduo Wang
user
Tianduo
66a740d7577ca69aa4c1c4a9
Shichen Li
66a740d7577ca69aa4c1c4aa
Wei Lu
admin_assigned
2024-07-30T08:39:47.467Z
6447f0ed19538c015b27ecb8
/avatars/aae591acfe36e3297d57103d1bda12cf.svg
Wei Lu
user
weilu
Self-Training with Direct Preference Optimization Improves
  Chain-of-Thought Reasoning
Effective training of language models (LMs) for mathematical reasoning tasks
demands high-quality supervised fine-tuning data. Besides obtaining annotations
from human experts, a common alternative is sampling from larger and more
powerful LMs. However, this knowledge distillation approach can be costly and
unstable, particularly when relying on closed-source, proprietary LMs like
GPT-4, whose behaviors are often unpredictable. In this work, we demonstrate
that the reasoning abilities of small-scale LMs can be enhanced through
self-training, a process where models learn from their own outputs. We also
show that the conventional self-training can be further augmented by a
preference learning algorithm called Direct Preference Optimization (DPO). By
integrating DPO into self-training, we leverage preference data to guide LMs
towards more accurate and diverse chain-of-thought reasoning. We evaluate our
method across various mathematical reasoning tasks using different base models.
Our experiments show that this approach not only improves LMs' reasoning
performance but also offers a more cost-effective and scalable solution
compared to relying on large proprietary LMs.
Enhancing small-scale language models for mathematical reasoning through self-training and preference learning, resulting in improved performance and reduced costs compared to large proprietary models.
language models fine-tuning knowledge distillation self-training Direct Preference Optimization preference learning chain-of-thought reasoning mathematical reasoning tasks