Direct Language Model Alignment from Online AI Feedback
Direct alignment from preferences (DAP) methods, such as DPO, have recently
emerged as efficient alternatives to reinforcement learning from human feedback
(RLHF), that do not require a separate reward model. However, the preference
datasets used in DAP methods are usually collected ahead of training and never
updated, thus the feedback is purely offline. Moreover, responses in these
datasets are often sampled from a language model distinct from the one being
aligned, and since the model evolves over training, the alignment phase is
inevitably off-policy. In this study, we posit that online feedback is key and
improves DAP methods. Our method, online AI feedback (OAIF), uses an LLM as
annotator: on each training iteration, we sample two responses from the current
model and prompt the LLM annotator to choose which one is preferred, thus
providing online feedback. Despite its simplicity, we demonstrate via human
evaluation in several tasks that OAIF outperforms both offline DAP and RLHF
methods. We further show that the feedback leveraged in OAIF is easily
controllable, via instruction prompts to the LLM annotator.
65c43fca7273e436da4ace13
Shangmin Guo
claimed_verified
2024-02-08T11:52:50.223Z
64406aac7841867cd5b5b9fa
https://cdn-avatars.huggingface.co/v1/production/uploads/64406aac7841867cd5b5b9fa/hPPiqtbYv-42q8LOwB22p.png
Shawn Guo
user
Shawn-Guo
65c43fca7273e436da4ace14
Biao Zhang
claimed_verified
2024-02-09T08:05:51.869Z
6346a600bcb5e67902ac6f44
https://cdn-avatars.huggingface.co/v1/production/uploads/6346a600bcb5e67902ac6f44/Kw-Ucmxx5S9WdlYDxQQ-z.jpeg
Biao Zhang
user
bzhanggo
65c43fca7273e436da4ace15
Tianlin Liu
admin_assigned
2024-02-08T11:10:49.055Z
64268d8d2f16fd599d9e9576
https://cdn-avatars.huggingface.co/v1/production/uploads/64268d8d2f16fd599d9e9576/yTWuz4wQbXD6QlovBY6-M.jpeg
Tianlin Liu
user
tianlinliu0121
65c43fca7273e436da4ace16
Tianqi Liu
claimed_verified
2024-02-29T08:40:54.017Z
646784af696e7355f5d5001f
/avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
Tianqi Liu
user
TianqiLiuAI
65c43fca7273e436da4ace17
Misha Khalman
admin_assigned
2024-02-08T11:11:00.617Z
65c2ad2790703d2dbcf79471
https://cdn-avatars.huggingface.co/v1/production/uploads/65c2ad2790703d2dbcf79471/H15_ASSI7fCFDCDF4xgIb.jpeg
Misha Khalman
user
khalman
65c43fca7273e436da4ace18
Felipe Llinares
65c43fca7273e436da4ace19
Alexandre Rame
admin_assigned
2024-02-08T11:11:13.950Z
63c94ede00104ea998de19a6
/avatars/273959d87f0c67747588cf0700d64039.svg
Alexandre Rame
user
alexrame
65c43fca7273e436da4ace1a
Thomas Mesnard
65c43fca7273e436da4ace1b
Yao Zhao
65c43fca7273e436da4ace1c
Bilal Piot
65c43fca7273e436da4ace1d
Johan Ferret
admin_assigned
2024-02-08T11:11:45.967Z
65afb7dbdd6bdfd73cd8e609
/avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
Johan Ferret
user
ferretj
65c43fca7273e436da4ace1e
Mathieu Blondel
claimed_verified
2024-08-29T16:46:53.081Z
66d093b681e0683bca48bed6
/avatars/cc1fbccb0b6aa93d648bcbdf9c3a35e1.svg
Mathieu Blondel
user
mblondel
Direct Language Model Alignment from Online AI Feedback
Direct alignment from preferences (DAP) methods, such as DPO, have recently
emerged as efficient alternatives to reinforcement learning from human feedback
(RLHF), that do not require a separate reward model. However, the preference
datasets used in DAP methods are usually collected ahead of training and never
updated, thus the feedback is purely offline. Moreover, responses in these
datasets are often sampled from a language model distinct from the one being
aligned, and since the model evolves over training, the alignment phase is
inevitably off-policy. In this study, we posit that online feedback is key and
improves DAP methods. Our method, online AI feedback (OAIF), uses an LLM as
annotator: on each training iteration, we sample two responses from the current
model and prompt the LLM annotator to choose which one is preferred, thus
providing online feedback. Despite its simplicity, we demonstrate via human
evaluation in several tasks that OAIF outperforms both offline DAP and RLHF
methods. We further show that the feedback leveraged in OAIF is easily
controllable, via instruction prompts to the LLM annotator.
Online AI feedback improves direct alignment from preferences methods by providing real-time feedback from a model, outperforming both offline DAP and RLHF methods.
direct alignment from preferences DPO reinforcement learning from human feedback preference datasets offline feedback on-policy off-policy online AI feedback LLM annotator human evaluation