Deep Bayesian Active Learning for Preference Modeling in Large Language
  Models
Leveraging human preferences for steering the behavior of Large Language
Models (LLMs) has demonstrated notable success in recent years. Nonetheless,
data selection and labeling are still a bottleneck for these systems,
particularly at large scale. Hence, selecting the most informative points for
acquiring human feedback may considerably reduce the cost of preference
labeling and unleash the further development of LLMs. Bayesian Active Learning
provides a principled framework for addressing this challenge and has
demonstrated remarkable success in diverse settings. However, previous attempts
to employ it for Preference Modeling did not meet such expectations. In this
work, we identify that naive epistemic uncertainty estimation leads to the
acquisition of redundant samples. We address this by proposing the Bayesian
Active Learner for Preference Modeling (BAL-PM), a novel stochastic acquisition
policy that not only targets points of high epistemic uncertainty according to
the preference model but also seeks to maximize the entropy of the acquired
prompt distribution in the feature space spanned by the employed LLM. Notably,
our experiments demonstrate that BAL-PM requires 33% to 68% fewer preference
labels in two popular human preference datasets and exceeds previous stochastic
Bayesian acquisition policies.
6671b1cf059c8f7fc78187a5
Luckeciano C. Melo
6671b1cf059c8f7fc78187a6
Panagiotis Tigas
6671b1cf059c8f7fc78187a7
Alessandro Abate
admin_assigned
2024-06-20T10:30:27.816Z
646723426192d3914221997b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/TL7QhvPJEMEch3a6f8cEr.jpeg
Alessandro Abate
user
SteamDrake
6671b1cf059c8f7fc78187a8
Yarin Gal
Deep Bayesian Active Learning for Preference Modeling in Large Language
  Models
Leveraging human preferences for steering the behavior of Large Language
Models (LLMs) has demonstrated notable success in recent years. Nonetheless,
data selection and labeling are still a bottleneck for these systems,
particularly at large scale. Hence, selecting the most informative points for
acquiring human feedback may considerably reduce the cost of preference
labeling and unleash the further development of LLMs. Bayesian Active Learning
provides a principled framework for addressing this challenge and has
demonstrated remarkable success in diverse settings. However, previous attempts
to employ it for Preference Modeling did not meet such expectations. In this
work, we identify that naive epistemic uncertainty estimation leads to the
acquisition of redundant samples. We address this by proposing the Bayesian
Active Learner for Preference Modeling (BAL-PM), a novel stochastic acquisition
policy that not only targets points of high epistemic uncertainty according to
the preference model but also seeks to maximize the entropy of the acquired
prompt distribution in the feature space spanned by the employed LLM. Notably,
our experiments demonstrate that BAL-PM requires 33% to 68% fewer preference
labels in two popular human preference datasets and exceeds previous stochastic
Bayesian acquisition policies.
The Bayesian Active Learner for Preference Modeling (BAL-PM) reduces the need for human preference labels by strategically selecting informative samples, outperforming previous methods.
Large Language Models (LLMs) Bayesian Active Learning preference modeling epistemic uncertainty entropy feature space preference labels