BOND: Aligning LLMs with Best-of-N Distillation
Reinforcement learning from human feedback (RLHF) is a key driver of quality
and safety in state-of-the-art large language models. Yet, a surprisingly
simple and strong inference-time strategy is Best-of-N sampling that selects
the best generation among N candidates. In this paper, we propose Best-of-N
Distillation (BOND), a novel RLHF algorithm that seeks to emulate Best-of-N but
without its significant computational overhead at inference time. Specifically,
BOND is a distribution matching algorithm that forces the distribution of
generations from the policy to get closer to the Best-of-N distribution. We use
the Jeffreys divergence (a linear combination of forward and backward KL) to
balance between mode-covering and mode-seeking behavior, and derive an
iterative formulation that utilizes a moving anchor for efficiency. We
demonstrate the effectiveness of our approach and several design choices
through experiments on abstractive summarization and Gemma models. Aligning
Gemma policies with BOND outperforms other RLHF algorithms by improving results
on several benchmarks.
669f50b24b821f5697c3a895
Pier Giuseppe Sessa
admin_assigned
2024-07-23T08:46:21.684Z
667a7876becec8fc5112cf1f
/avatars/6874f0e5a7edca4145c57d7908f89840.svg
Pier Giuseppe Sessa
user
piergs
669f50b24b821f5697c3a896
Robert Dadashi
admin_assigned
2024-07-23T08:46:27.635Z
65fc7c0da57b88a765aea493
/avatars/440f88769da1113d6158fe7e0514ead3.svg
Robert Dadashi
user
ddsh
669f50b24b821f5697c3a897
Léonard Hussenot
admin_assigned
2024-07-23T08:46:36.305Z
667a7b2e1764931987803b46
/avatars/da2ee162f33c681e3d1c4d1e6d44dcbd.svg
Léonard Hussenot
user
leonardhussenot
669f50b24b821f5697c3a898
Johan Ferret
admin_assigned
2024-07-23T08:46:42.102Z
65afb7dbdd6bdfd73cd8e609
/avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
Johan Ferret
user
ferretj
669f50b24b821f5697c3a899
Nino Vieillard
669f50b24b821f5697c3a89a
Alexandre Ramé
admin_assigned
2024-07-23T08:46:55.023Z
63c94ede00104ea998de19a6
/avatars/273959d87f0c67747588cf0700d64039.svg
Alexandre Rame
user
alexrame
669f50b24b821f5697c3a89b
Bobak Shariari
669f50b24b821f5697c3a89c
Sarah Perrin
admin_assigned
2024-07-23T10:04:32.674Z
66328157b270ae503e91339b
/avatars/ea7a52060f5360f523ca28e137e85e33.svg
Sarah Perrin
user
Sper42
669f50b24b821f5697c3a89d
Abe Friesen
669f50b24b821f5697c3a89e
Geoffrey Cideron
669f50b24b821f5697c3a89f
Sertan Girgin
669f50b24b821f5697c3a8a0
Piotr Stanczyk
669f50b24b821f5697c3a8a1
Andrea Michi
admin_assigned
2024-07-23T10:08:27.975Z
66466ef3b080e0f563dd2372
https://cdn-avatars.huggingface.co/v1/production/uploads/66466ef3b080e0f563dd2372/wOMZgsd8r28iwAvqmYQRI.jpeg
Andrea Michi
user
andreamichi
669f50b24b821f5697c3a8a2
Danila Sinopalnikov
669f50b24b821f5697c3a8a3
Sabela Ramos
669f50b24b821f5697c3a8a4
Amélie Héliou
669f50b24b821f5697c3a8a5
Aliaksei Severyn
669f50b24b821f5697c3a8a6
Matt Hoffman
669f50b24b821f5697c3a8a7
Nikola Momchev
669f50b24b821f5697c3a8a8
Olivier Bachem
admin_assigned
2024-07-23T10:04:40.875Z
667aac871bffb68706b4f62c
/avatars/2e2f2431f03a368f604c992d0d7ca57a.svg
Olivier Bachem
user
bachem
BOND: Aligning LLMs with Best-of-N Distillation
Reinforcement learning from human feedback (RLHF) is a key driver of quality
and safety in state-of-the-art large language models. Yet, a surprisingly
simple and strong inference-time strategy is Best-of-N sampling that selects
the best generation among N candidates. In this paper, we propose Best-of-N
Distillation (BOND), a novel RLHF algorithm that seeks to emulate Best-of-N but
without its significant computational overhead at inference time. Specifically,
BOND is a distribution matching algorithm that forces the distribution of
generations from the policy to get closer to the Best-of-N distribution. We use
the Jeffreys divergence (a linear combination of forward and backward KL) to
balance between mode-covering and mode-seeking behavior, and derive an
iterative formulation that utilizes a moving anchor for efficiency. We
demonstrate the effectiveness of our approach and several design choices
through experiments on abstractive summarization and Gemma models. Aligning
Gemma policies with BOND outperforms other RLHF algorithms by improving results
on several benchmarks.
BOND, a novel RLHF algorithm, distills the Best-of-N sampling strategy without significant computational overhead, enhancing the quality of generative language models.
Reinforcement learning from human feedback RLHF Best-of-N sampling Best-of-N Distillation BOND distribution matching Jeffreys divergence forward KL backward KL mode-covering mode-seeking abstractive summarization Gemma models