Self-Consistency Preference Optimization
Self-alignment, whereby models learn to improve themselves without human
annotation, is a rapidly growing research area. However, existing techniques
often fail to improve complex reasoning tasks due to the difficulty of
assigning correct rewards. An orthogonal approach that is known to improve
correctness is self-consistency, a method applied at inference time based on
multiple sampling in order to find the most consistent answer. In this work, we
extend the self-consistency concept to help train models. We thus introduce
self-consistency preference optimization (ScPO), which iteratively trains
consistent answers to be preferred over inconsistent ones on unsupervised new
problems. We show ScPO leads to large improvements over conventional reward
model training on reasoning tasks such as GSM8K and MATH, closing the gap with
supervised training with gold answers or preferences, and that combining ScPO
with standard supervised learning improves results even further. On ZebraLogic,
ScPO finetunes Llama-3 8B to be superior to Llama-3 70B, Gemma-2 27B, and
Claude-3 Haiku.
672cbe66a1e855c3297c98d4
Archiki Prasad
claimed_verified
2024-11-15T09:33:11.771Z
607aeae5d2cd8c150e6ae074
/avatars/a087743b98b6fe2181283a9610db4ec4.svg
Archiki Prasad
user
archiki
672cbe66a1e855c3297c98d5
Weizhe Yuan
672cbe66a1e855c3297c98d6
Richard Yuanzhe Pang
672cbe66a1e855c3297c98d7
Jing Xu
672cbe66a1e855c3297c98d8
Maryam Fazel-Zarandi
672cbe66a1e855c3297c98d9
Mohit Bansal
672cbe66a1e855c3297c98da
Sainbayar Sukhbaatar
claimed_verified
2025-06-05T07:30:47.108Z
66a8611eb51510d82ed54231
/avatars/ad559e774fee4914091b82c9831ae2a2.svg
Sainbayar Sukhbaatar
user
sainbar
672cbe66a1e855c3297c98db
Jason Weston
672cbe66a1e855c3297c98dc
Jane Yu
Self-Consistency Preference Optimization
Self-alignment, whereby models learn to improve themselves without human
annotation, is a rapidly growing research area. However, existing techniques
often fail to improve complex reasoning tasks due to the difficulty of
assigning correct rewards. An orthogonal approach that is known to improve
correctness is self-consistency, a method applied at inference time based on
multiple sampling in order to find the most consistent answer. In this work, we
extend the self-consistency concept to help train models. We thus introduce
self-consistency preference optimization (ScPO), which iteratively trains
consistent answers to be preferred over inconsistent ones on unsupervised new
problems. We show ScPO leads to large improvements over conventional reward
model training on reasoning tasks such as GSM8K and MATH, closing the gap with
supervised training with gold answers or preferences, and that combining ScPO
with standard supervised learning improves results even further. On ZebraLogic,
ScPO finetunes Llama-3 8B to be superior to Llama-3 70B, Gemma-2 27B, and
Claude-3 Haiku.
Self-consistency preference optimization (ScPO) enhances model training by iteratively preferring consistent answers, improving performance on reasoning tasks and outperforming larger models on specific benchmarks.
self-alignment self-consistency preference optimization ScPO unsupervised training reasoning tasks GSM8K MATH ZebraLogic Llama-3 Gemma-2 Claude-3 Haiku