Statistical Rejection Sampling Improves Preference Optimization
Improving the alignment of language models with human preferences remains an
active research challenge. Previous approaches have primarily utilized
Reinforcement Learning from Human Feedback (RLHF) via online RL methods such as
Proximal Policy Optimization (PPO). Recently, offline methods such as Sequence
Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have
emerged as attractive alternatives, offering improvements in stability and
scalability while maintaining competitive performance. SLiC refines its loss
function using sequence pairs sampled from a supervised fine-tuned (SFT)
policy, while DPO directly optimizes language models based on preference data,
foregoing the need for a separate reward model. However, the maximum likelihood
estimator (MLE) of the target optimal policy requires labeled preference pairs
sampled from that policy. DPO's lack of a reward model constrains its ability
to sample preference pairs from the optimal policy, and SLiC is restricted to
sampling preference pairs only from the SFT policy. To address these
limitations, we introduce a novel approach called Statistical Rejection
Sampling Optimization (RSO) that aims to source preference data from the target
optimal policy using rejection sampling, enabling a more accurate estimation of
the optimal policy. We also propose a unified framework that enhances the loss
functions used in both SLiC and DPO from a preference modeling standpoint.
Through extensive experiments across three diverse tasks, we demonstrate that
RSO consistently outperforms both SLiC and DPO on evaluations from both Large
Language Model (LLM) and human raters.
65026cf6e751d03da934a16f
Tianqi Liu
admin_assigned
2023-09-14T07:35:32.897Z
646784af696e7355f5d5001f
/avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
Tianqi Liu
user
TianqiLiuAI
65026cf6e751d03da934a170
Yao Zhao
65026cf6e751d03da934a171
Rishabh Joshi
65026cf6e751d03da934a172
Misha Khalman
claimed_verified
2024-02-07T20:41:23.216Z
65c2ad2790703d2dbcf79471
https://cdn-avatars.huggingface.co/v1/production/uploads/65c2ad2790703d2dbcf79471/H15_ASSI7fCFDCDF4xgIb.jpeg
Misha Khalman
user
khalman
65026cf6e751d03da934a173
Mohammad Saleh
65026cf6e751d03da934a174
Peter J. Liu
admin_assigned
2023-09-14T07:37:13.544Z
62ded294dcdc9043efaa4fc3
/avatars/1294078b8cd5c1901e4021253d877357.svg
Peter J Liu
user
peterjliu
65026cf6e751d03da934a175
Jialu Liu
admin_assigned
2023-09-14T07:37:20.469Z
64acb90a5d48838462da09ae
/avatars/dc87416dcb2806a6e0d60bd636024978.svg
liujialu
user
Jialuliu
Statistical Rejection Sampling Improves Preference Optimization
Improving the alignment of language models with human preferences remains an
active research challenge. Previous approaches have primarily utilized
Reinforcement Learning from Human Feedback (RLHF) via online RL methods such as
Proximal Policy Optimization (PPO). Recently, offline methods such as Sequence
Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have
emerged as attractive alternatives, offering improvements in stability and
scalability while maintaining competitive performance. SLiC refines its loss
function using sequence pairs sampled from a supervised fine-tuned (SFT)
policy, while DPO directly optimizes language models based on preference data,
foregoing the need for a separate reward model. However, the maximum likelihood
estimator (MLE) of the target optimal policy requires labeled preference pairs
sampled from that policy. DPO's lack of a reward model constrains its ability
to sample preference pairs from the optimal policy, and SLiC is restricted to
sampling preference pairs only from the SFT policy. To address these
limitations, we introduce a novel approach called Statistical Rejection
Sampling Optimization (RSO) that aims to source preference data from the target
optimal policy using rejection sampling, enabling a more accurate estimation of
the optimal policy. We also propose a unified framework that enhances the loss
functions used in both SLiC and DPO from a preference modeling standpoint.
Through extensive experiments across three diverse tasks, we demonstrate that
RSO consistently outperforms both SLiC and DPO on evaluations from both Large
Language Model (LLM) and human raters.
A novel approach called Statistical Rejection Sampling Optimization (RSO) enhances preference data sourcing for aligning language models with human preferences, outperforming previous methods like SLiC and DPO.
Reinforcement Learning from Human Feedback (RLHF) Proximal Policy Optimization (PPO) Sequence Likelihood Calibration (SLiC) Direct Preference Optimization (DPO) maximum likelihood estimator (MLE) target optimal policy rejection sampling Statistical Rejection Sampling Optimization (RSO) preference modeling Large Language Model (LLM)