New Desiderata for Direct Preference Optimization
Large language models in the past have typically relied on some form of
reinforcement learning with human feedback (RLHF) to better align model
responses with human preferences. However, because of oft-observed
instabilities when implementing these RLHF pipelines, various
reparameterization techniques have recently been introduced to sidestep the
need for separately learning an RL reward model. Instead, directly fine-tuning
for human preferences is achieved via the minimization of a single closed-form
training objective, a process originally referred to as direct preference
optimization (DPO) and followed by several notable descendants. Although
effective in certain real-world settings, we introduce new evaluation criteria
that serve to highlight unresolved shortcomings in the ability of existing DPO
methods to interpolate between a pre-trained reference model and empirical
measures of human preferences, as well as unavoidable trade-offs in how low-
and high-quality responses are regularized and constraints are handled. Our
insights then motivate an alternative DPO-like loss that provably mitigates
these limitations. Empirical results serve to corroborate notable aspects of
our analyses.
6694d14ccab4aab691fa1c71
Xiangkun Hu
6694d14ccab4aab691fa1c72
Tong He
claimed_verified
2024-07-15T08:58:03.349Z
63184606ab1bf092d6f059be
/avatars/b49d125d112a01106b48a2fcda481c3e.svg
Tong He
user
hetong007
6694d14ccab4aab691fa1c73
David Wipf
New Desiderata for Direct Preference Optimization
Large language models in the past have typically relied on some form of
reinforcement learning with human feedback (RLHF) to better align model
responses with human preferences. However, because of oft-observed
instabilities when implementing these RLHF pipelines, various
reparameterization techniques have recently been introduced to sidestep the
need for separately learning an RL reward model. Instead, directly fine-tuning
for human preferences is achieved via the minimization of a single closed-form
training objective, a process originally referred to as direct preference
optimization (DPO) and followed by several notable descendants. Although
effective in certain real-world settings, we introduce new evaluation criteria
that serve to highlight unresolved shortcomings in the ability of existing DPO
methods to interpolate between a pre-trained reference model and empirical
measures of human preferences, as well as unavoidable trade-offs in how low-
and high-quality responses are regularized and constraints are handled. Our
insights then motivate an alternative DPO-like loss that provably mitigates
these limitations. Empirical results serve to corroborate notable aspects of
our analyses.
Current DPO methods face limitations in interpolating between pre-trained models and human preferences, so an alternative DPO loss is introduced to address these issues.
reinforcement learning with human feedback RLHF reparameterization techniques direct preference optimization DPO training objective human preferences pre-trained reference model low-quality responses high-quality responses constraints