Hybrid Preferences: Learning to Route Instances for Human vs. AI
  Feedback
Learning from human feedback has enabled the alignment of language models
(LMs) with human preferences. However, directly collecting human preferences
can be expensive, time-consuming, and can have high variance. An appealing
alternative is to distill preferences from LMs as a source of synthetic
annotations as they are more consistent, cheaper, and scale better than human
annotation; however, they are also prone to biases and errors. In this work, we
introduce a routing framework that combines inputs from humans and LMs to
achieve better annotation quality, while reducing the total cost of human
annotation. The crux of our approach is to identify preference instances that
will benefit from human annotations. We formulate this as an optimization
problem: given a preference dataset and an evaluation metric, we train a
performance prediction model to predict a reward model's performance on an
arbitrary combination of human and LM annotations and employ a routing strategy
that selects a combination that maximizes predicted performance. We train the
performance prediction model on MultiPref, a new preference dataset with 10K
instances paired with human and LM labels. We show that the selected hybrid
mixture of LM and direct human preferences using our routing framework achieves
better reward model performance compared to using either one exclusively. We
simulate selective human preference collection on three other datasets and show
that our method generalizes well to all three. We analyze features from the
routing model to identify characteristics of instances that can benefit from
human feedback, e.g., prompts with a moderate safety concern or moderate intent
complexity. We release the dataset, annotation platform, and source code used
in this study to foster more efficient and accurate preference collection in
the future.
671ed82fb91f5a4a244a9d87
Lester James V. Miranda
claimed_verified
2024-10-28T10:33:55.165Z
634e20a0c1ce28f1de920cc4
https://cdn-avatars.huggingface.co/v1/production/uploads/1666064515342-noauth.jpeg
Lj V. Miranda
user
ljvmiranda921
671ed82fb91f5a4a244a9d88
Yizhong Wang
admin_assigned
2024-10-28T15:40:01.833Z
6269be67d1ac0cde592aba29
https://cdn-avatars.huggingface.co/v1/production/uploads/6269be67d1ac0cde592aba29/EP3zZbD7-jWK9ITvHGbnZ.jpeg
Yizhong Wang
user
yizhongw
671ed82fb91f5a4a244a9d89
Yanai Elazar
admin_assigned
2024-10-28T15:40:23.763Z
623ca115a795593324c4353f
/avatars/bf11fe728df2786d52ed4d2de12b48d3.svg
Yanai Elazar
user
yanaiela
671ed82fb91f5a4a244a9d8a
Sachin Kumar
admin_assigned
2024-10-28T15:41:48.584Z
63f24d2d7ddf724fbcc0ea9c
/avatars/3e24c1aa9c1b4066d2dd56aeb4b0f62e.svg
sachin kumar
user
sachinkumar
671ed82fb91f5a4a244a9d8b
Valentina Pyatkin
admin_assigned
2024-10-28T15:41:55.289Z
6556cff80e7a7067a934445f
https://cdn-avatars.huggingface.co/v1/production/uploads/6556cff80e7a7067a934445f/PoT7qQ6tVqLGGrYvGBbr8.jpeg
Valentina Pyatkin
user
valpy
671ed82fb91f5a4a244a9d8c
Faeze Brahman
admin_assigned
2024-10-28T15:42:00.686Z
65282b8d578679aac7888aec
https://cdn-avatars.huggingface.co/v1/production/uploads/65282b8d578679aac7888aec/dibBkhH-z1c70mJZZxJ7u.jpeg
Faeze Brahman
user
faezeb
671ed82fb91f5a4a244a9d8d
Noah A. Smith
671ed82fb91f5a4a244a9d8e
Hannaneh Hajishirzi
671ed82fb91f5a4a244a9d8f
Pradeep Dasigi
admin_assigned
2024-10-28T15:42:06.735Z
6408fcc93461c51cf735a61e
/avatars/619f3653911d111f046a5a6c30fc8319.svg
Pradeep Dasigi
user
pradeepd
Hybrid Preferences: Learning to Route Instances for Human vs. AI
  Feedback
Learning from human feedback has enabled the alignment of language models
(LMs) with human preferences. However, directly collecting human preferences
can be expensive, time-consuming, and can have high variance. An appealing
alternative is to distill preferences from LMs as a source of synthetic
annotations as they are more consistent, cheaper, and scale better than human
annotation; however, they are also prone to biases and errors. In this work, we
introduce a routing framework that combines inputs from humans and LMs to
achieve better annotation quality, while reducing the total cost of human
annotation. The crux of our approach is to identify preference instances that
will benefit from human annotations. We formulate this as an optimization
problem: given a preference dataset and an evaluation metric, we train a
performance prediction model to predict a reward model's performance on an
arbitrary combination of human and LM annotations and employ a routing strategy
that selects a combination that maximizes predicted performance. We train the
performance prediction model on MultiPref, a new preference dataset with 10K
instances paired with human and LM labels. We show that the selected hybrid
mixture of LM and direct human preferences using our routing framework achieves
better reward model performance compared to using either one exclusively. We
simulate selective human preference collection on three other datasets and show
that our method generalizes well to all three. We analyze features from the
routing model to identify characteristics of instances that can benefit from
human feedback, e.g., prompts with a moderate safety concern or moderate intent
complexity. We release the dataset, annotation platform, and source code used
in this study to foster more efficient and accurate preference collection in
the future.
A routing framework that combines human and synthetic annotations using a performance prediction model improves reward model performance and reduces human annotation costs.
performance prediction model reward model routing framework synthetic annotations preference dataset human annotation preference collection annotation quality safety concern intent complexity