BPO: Supercharging Online Preference Learning by Adhering to the
  Proximity of Behavior LLM
Direct alignment from preferences (DAP) has emerged as a promising paradigm
for aligning large language models (LLMs) to human desiderata from
pre-collected, offline preference datasets. While recent studies indicate that
existing offline DAP methods can directly benefit from online training samples,
we highlight the need to develop specific online DAP algorithms to fully
harness the power of online training. Specifically, we identify that the
learned LLM should adhere to the proximity of the behavior LLM, which collects
the training samples. To this end, we propose online Preference Optimization in
proximity to the Behavior LLM (BPO), emphasizing the importance of constructing
a proper trust region for LLM alignment.
  We conduct extensive experiments to validate the effectiveness and
applicability of our approach by integrating it with various DAP methods,
resulting in significant performance improvements across a wide range of tasks
when training with the same amount of preference data. Even when only
introducing one additional data collection phase, our online BPO improves its
offline DAP baseline from 72.0% to 80.2% on TL;DR and from 82.2% to 89.1% on
Anthropic Helpfulness in terms of win rate against human reference text.
667266a20f6ec76f334bf263
Wenda Xu
667266a20f6ec76f334bf264
Jiachen Li
claimed_verified
2024-06-19T07:37:31.704Z
640d7ec5fdeaae13907fc488
/avatars/acf43ea155105a51c8612dacc4725091.svg
Jiachen Li
user
jiachenli-ucsb
667266a20f6ec76f334bf265
William Yang Wang
admin_assigned
2024-06-20T10:19:15.810Z
6476607efb22e3b77f3f1452
/avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
William Yang Wang
user
wangwilliamyang
667266a20f6ec76f334bf266
Lei Li
BPO: Supercharging Online Preference Learning by Adhering to the
  Proximity of Behavior LLM
Direct alignment from preferences (DAP) has emerged as a promising paradigm
for aligning large language models (LLMs) to human desiderata from
pre-collected, offline preference datasets. While recent studies indicate that
existing offline DAP methods can directly benefit from online training samples,
we highlight the need to develop specific online DAP algorithms to fully
harness the power of online training. Specifically, we identify that the
learned LLM should adhere to the proximity of the behavior LLM, which collects
the training samples. To this end, we propose online Preference Optimization in
proximity to the Behavior LLM (BPO), emphasizing the importance of constructing
a proper trust region for LLM alignment.
  We conduct extensive experiments to validate the effectiveness and
applicability of our approach by integrating it with various DAP methods,
resulting in significant performance improvements across a wide range of tasks
when training with the same amount of preference data. Even when only
introducing one additional data collection phase, our online BPO improves its
offline DAP baseline from 72.0% to 80.2% on TL;DR and from 82.2% to 89.1% on
Anthropic Helpfulness in terms of win rate against human reference text.
Online Preference Optimization in proximity to the Behavior LLM (BPO) enhances the alignment of large language models to human preferences by integrating offline DAP methods with online training samples, leading to significant performance improvements.
Direct alignment from preferences DAP large language models LLMs offline preference datasets online DAP algorithms learned LLM behavior LLM trust region preference optimization TL;DR Anthropic Helpfulness win rate