SLiC-HF: Sequence Likelihood Calibration with Human Feedback
Learning from human feedback has been shown to be effective at aligning
language models with human preferences. Past work has often relied on
Reinforcement Learning from Human Feedback (RLHF), which optimizes the language
model using reward scores assigned from a reward model trained on human
preference data. In this work we show how the recently introduced Sequence
Likelihood Calibration (SLiC), can also be used to effectively learn from human
preferences (SLiC-HF). Furthermore, we demonstrate this can be done with human
feedback data collected for a different model, similar to off-policy, offline
RL data. Automatic and human evaluation experiments on the TL;DR summarization
task show that SLiC-HF significantly improves supervised fine-tuning baselines.
Furthermore, SLiC-HF presents a competitive alternative to the PPO RLHF
implementation used in past work while being much simpler to implement, easier
to tune and more computationally efficient in practice.
6465885e34acce85aea4dde7
Yao Zhao
6465885e34acce85aea4dde8
Rishabh Joshi
6465885e34acce85aea4dde9
Tianqi Liu
extracted_confirmed
2023-05-19T14:17:57.179Z
646784af696e7355f5d5001f
/avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
Tianqi Liu
user
TianqiLiuAI
6465885e34acce85aea4ddea
Misha Khalman
claimed_verified
2024-02-07T20:41:29.156Z
65c2ad2790703d2dbcf79471
https://cdn-avatars.huggingface.co/v1/production/uploads/65c2ad2790703d2dbcf79471/H15_ASSI7fCFDCDF4xgIb.jpeg
Misha Khalman
user
khalman
6465885e34acce85aea4ddeb
Mohammad Saleh
6465885e34acce85aea4ddec
Peter J. Liu
extracted_confirmed
2023-05-18T02:08:56.065Z
62ded294dcdc9043efaa4fc3
/avatars/1294078b8cd5c1901e4021253d877357.svg
Peter J Liu
user
peterjliu
SLiC-HF: Sequence Likelihood Calibration with Human Feedback
Learning from human feedback has been shown to be effective at aligning
language models with human preferences. Past work has often relied on
Reinforcement Learning from Human Feedback (RLHF), which optimizes the language
model using reward scores assigned from a reward model trained on human
preference data. In this work we show how the recently introduced Sequence
Likelihood Calibration (SLiC), can also be used to effectively learn from human
preferences (SLiC-HF). Furthermore, we demonstrate this can be done with human
feedback data collected for a different model, similar to off-policy, offline
RL data. Automatic and human evaluation experiments on the TL;DR summarization
task show that SLiC-HF significantly improves supervised fine-tuning baselines.
Furthermore, SLiC-HF presents a competitive alternative to the PPO RLHF
implementation used in past work while being much simpler to implement, easier
to tune and more computationally efficient in practice.
Sequence Likelihood Calibration (SLiC) is shown to be an effective and simpler alternative to Reinforcement Learning from Human Feedback (RLHF) for learning from human preferences in language models.
Reinforcement Learning from Human Feedback (RLHF) Sequence Likelihood Calibration (SLiC) human feedback human preferences PPO supervised fine-tuning