Direct Nash Optimization: Teaching Language Models to Self-Improve with
  General Preferences
This paper studies post-training large language models (LLMs) using
preference feedback from a powerful oracle to help a model iteratively improve
over itself. The typical approach for post-training LLMs involves Reinforcement
Learning from Human Feedback (RLHF), which traditionally separates reward
learning and subsequent policy optimization. However, such a reward
maximization approach is limited by the nature of "point-wise" rewards (such as
Bradley-Terry model), which fails to express complex intransitive or cyclic
preference relations. While advances on RLHF show reward learning and policy
optimization can be merged into a single contrastive objective for stability,
they yet still remain tethered to the reward maximization framework. Recently,
a new wave of research sidesteps the reward maximization presumptions in favor
of directly optimizing over "pair-wise" or general preferences. In this paper,
we introduce Direct Nash Optimization (DNO), a provable and scalable algorithm
that marries the simplicity and stability of contrastive learning with
theoretical generality from optimizing general preferences. Because DNO is a
batched on-policy algorithm using a regression-based objective, its
implementation is straightforward and efficient. Moreover, DNO enjoys monotonic
improvement across iterations that help it improve even over a strong teacher
(such as GPT-4). In our experiments, a resulting 7B parameter Orca-2.5 model
aligned by DNO achieves the state-of-the-art win-rate against GPT-4-Turbo of
33% on AlpacaEval 2.0 (even after controlling for response length), an absolute
gain of 26% (7% to 33%) over the initializing model. It outperforms models with
far more parameters, including Mistral Large, Self-Rewarding LM (70B
parameters), and older versions of GPT-4.
66135070174b378a723d9803
Corby Rosset
extracted_pending
2024-04-08T02:03:29.225Z
64ec43147e2ec711a761c594
/avatars/e4fc3de141ff026eee6eaa0b91d34ff1.svg
corby
user
corbyrosset
66135070174b378a723d9804
Ching-An Cheng
66135070174b378a723d9805
Arindam Mitra
66135070174b378a723d9806
Michael Santacroce
66135070174b378a723d9807
Ahmed Awadallah
66135070174b378a723d9808
Tengyang Xie
Direct Nash Optimization: Teaching Language Models to Self-Improve with
  General Preferences
This paper studies post-training large language models (LLMs) using
preference feedback from a powerful oracle to help a model iteratively improve
over itself. The typical approach for post-training LLMs involves Reinforcement
Learning from Human Feedback (RLHF), which traditionally separates reward
learning and subsequent policy optimization. However, such a reward
maximization approach is limited by the nature of "point-wise" rewards (such as
Bradley-Terry model), which fails to express complex intransitive or cyclic
preference relations. While advances on RLHF show reward learning and policy
optimization can be merged into a single contrastive objective for stability,
they yet still remain tethered to the reward maximization framework. Recently,
a new wave of research sidesteps the reward maximization presumptions in favor
of directly optimizing over "pair-wise" or general preferences. In this paper,
we introduce Direct Nash Optimization (DNO), a provable and scalable algorithm
that marries the simplicity and stability of contrastive learning with
theoretical generality from optimizing general preferences. Because DNO is a
batched on-policy algorithm using a regression-based objective, its
implementation is straightforward and efficient. Moreover, DNO enjoys monotonic
improvement across iterations that help it improve even over a strong teacher
(such as GPT-4). In our experiments, a resulting 7B parameter Orca-2.5 model
aligned by DNO achieves the state-of-the-art win-rate against GPT-4-Turbo of
33% on AlpacaEval 2.0 (even after controlling for response length), an absolute
gain of 26% (7% to 33%) over the initializing model. It outperforms models with
far more parameters, including Mistral Large, Self-Rewarding LM (70B
parameters), and older versions of GPT-4.
A Direct Nash Optimization (DNO) algorithm is introduced for post-training large language models using preference feedback, achieving state-of-the-art performance with polynomial parameters.
post-training large language models preference feedback Reinforcement Learning from Human Feedback (RLHF) point-wise rewards contrastive learning Direct Nash Optimization (DNO) batched on-policy algorithm polynomial parameters