RLCD: Reinforcement Learning from Contrast Distillation for Language
  Model Alignment
We propose Reinforcement Learning from Contrast Distillation (RLCD), a method
for aligning language models to follow natural language principles without
using human feedback. RLCD trains a preference model using simulated preference
pairs that contain both a high-quality and low-quality example, generated using
contrasting positive and negative prompts. The preference model is then used to
improve a base unaligned language model via reinforcement learning.
Empirically, RLCD outperforms RLAIF (Bai et al., 2022b) and context
distillation (Huang et al., 2022) baselines across three diverse alignment
tasks--harmlessness, helpfulness, and story outline generation--and on both 7B
and 30B model scales for preference data simulation.
64bf4334af05eb17c71c318e
Kevin Yang
claimed_verified
2023-07-26T08:44:01.730Z
645bf09b438d6cfbe1b0a73b
/avatars/6a9165fdaab493d692538005aeb63469.svg
Kevin Yang
user
yangkevin2
64bf4334af05eb17c71c318f
Dan Klein
admin_assigned
2023-07-25T09:00:43.344Z
632be88b3690fb57e70e0bf1
/avatars/74ff8f30b3662db2602495bdf493d397.svg
Dan Klein
user
danjklein
64bf4334af05eb17c71c3190
Asli Celikyilmaz
64bf4334af05eb17c71c3191
Nanyun Peng
claimed_verified
2023-07-26T16:15:17.996Z
64c13de54cbd12e168f09a58
https://cdn-avatars.huggingface.co/v1/production/uploads/64c13de54cbd12e168f09a58/Aw5BaNzf6uuwqQ333OQ5y.png
NANYUN (Violet) PENG
user
violetpeng
64bf4334af05eb17c71c3192
Yuandong Tian
admin_assigned
2023-07-25T09:01:30.739Z
6344cf73ee1504dbcd5bdfe7
/avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
Yuandong Tian
user
tydsh
RLCD: Reinforcement Learning from Contrast Distillation for Language
  Model Alignment
We propose Reinforcement Learning from Contrast Distillation (RLCD), a method
for aligning language models to follow natural language principles without
using human feedback. RLCD trains a preference model using simulated preference
pairs that contain both a high-quality and low-quality example, generated using
contrasting positive and negative prompts. The preference model is then used to
improve a base unaligned language model via reinforcement learning.
Empirically, RLCD outperforms RLAIF (Bai et al., 2022b) and context
distillation (Huang et al., 2022) baselines across three diverse alignment
tasks--harmlessness, helpfulness, and story outline generation--and on both 7B
and 30B model scales for preference data simulation.
Reinforcement Learning from Contrast Distillation (RLCD) aligns language models to natural language principles using simulated preference pairs without human feedback, outperforming existing methods across various alignment tasks.
Reinforcement Learning from Contrast Distillation (RLCD) preference model contrasting positive and negative prompts reinforcement learning RLAIF context distillation alignment tasks harmlessness helpfulness story outline generation