Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate
  Reward Hacking
Reward models play a key role in aligning language model applications towards
human preferences. However, this setup creates an incentive for the language
model to exploit errors in the reward model to achieve high estimated reward, a
phenomenon often termed reward hacking. A natural mitigation is to train
an ensemble of reward models, aggregating over model outputs to obtain a more
robust reward estimate. We explore the application of reward ensembles to
alignment at both training time (through reinforcement learning) and inference
time (through reranking). First, we show that reward models are
underspecified: reward models that perform similarly in-distribution can
yield very different rewards when used in alignment, due to distribution shift.
Second, underspecification results in overoptimization, where alignment to one
reward model does not improve reward as measured by another reward model
trained on the same data. Third, overoptimization is mitigated by the use of
reward ensembles, and ensembles that vary by their pretraining seeds
lead to better generalization than ensembles that differ only by their
fine-tuning seeds, with both outperforming individual reward models.
However, even pretrain reward ensembles do not eliminate reward hacking: we
show several qualitative reward hacking phenomena that are not mitigated by
ensembling because all reward models in the ensemble exhibit similar error
patterns.
657bcf8c538666d04ccc4723
Jacob Eisenstein
657bcf8c538666d04ccc4724
Chirag Nagpal
admin_assigned
2023-12-15T13:21:22.501Z
65080f431704c7eb0aaa0720
/avatars/b1f946b689fd9f99f9fff4df02bbc4f1.svg
Chirag Nagpal
user
chiragnagpal02
657bcf8c538666d04ccc4725
Alekh Agarwal
657bcf8c538666d04ccc4726
Ahmad Beirami
admin_assigned
2023-12-15T13:21:12.084Z
65482d45407bb19ff5adb8a4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_etKjaaubh3osxp3R2MP7.jpeg
Ahmad Beirami
user
beirami
657bcf8c538666d04ccc4727
Alex D'Amour
657bcf8c538666d04ccc4728
DJ Dvijotham
657bcf8c538666d04ccc4729
Adam Fisch
657bcf8c538666d04ccc472a
Katherine Heller
657bcf8c538666d04ccc472b
Stephen Pfohl
657bcf8c538666d04ccc472c
Deepak Ramachandran
657bcf8c538666d04ccc472d
Peter Shaw
657bcf8c538666d04ccc472e
Jonathan Berant
admin_assigned
2023-12-15T13:20:18.063Z
60811c9a4ad99100d63ce36c
https://cdn-avatars.huggingface.co/v1/production/uploads/1619119757620-60811c9a4ad99100d63ce36c.png
Jonathan Berant
user
joberant
Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate
  Reward Hacking
Reward models play a key role in aligning language model applications towards
human preferences. However, this setup creates an incentive for the language
model to exploit errors in the reward model to achieve high estimated reward, a
phenomenon often termed reward hacking. A natural mitigation is to train
an ensemble of reward models, aggregating over model outputs to obtain a more
robust reward estimate. We explore the application of reward ensembles to
alignment at both training time (through reinforcement learning) and inference
time (through reranking). First, we show that reward models are
underspecified: reward models that perform similarly in-distribution can
yield very different rewards when used in alignment, due to distribution shift.
Second, underspecification results in overoptimization, where alignment to one
reward model does not improve reward as measured by another reward model
trained on the same data. Third, overoptimization is mitigated by the use of
reward ensembles, and ensembles that vary by their pretraining seeds
lead to better generalization than ensembles that differ only by their
fine-tuning seeds, with both outperforming individual reward models.
However, even pretrain reward ensembles do not eliminate reward hacking: we
show several qualitative reward hacking phenomena that are not mitigated by
ensembling because all reward models in the ensemble exhibit similar error
patterns.
Reward ensembles improve language model alignment with human preferences but do not eliminate reward hacking.
reward hacking reinforcement learning reranking underspecified overoptimization pretraining seeds fine-tuning seeds