ODIN: Disentangled Reward Mitigates Hacking in RLHF
In this work, we study the issue of reward hacking on the response length, a
challenge emerging in Reinforcement Learning from Human Feedback (RLHF) on
LLMs. A well-formatted, verbose but less helpful response from the LLMs can
often deceive LLMs or even human evaluators to achieve high scores. The same
issue also holds for some reward models in RL. To address the challenges in
both training and evaluation, we establish a more reliable evaluation protocol
for comparing different training configurations, which inspects the trade-off
between LLM evaluation score and response length obtained by varying training
hyperparameters. Based on this evaluation, we conduct large-scale studies,
where the results shed insights into the efficacy of hyperparameters and tricks
used in RL on mitigating length bias. We further propose to improve the reward
model by jointly training two linear heads on shared feature representations to
predict the rewards, one trained to correlate with length, and the other
trained to decorrelate with length and therefore focus more on the actual
content. We then discard the length head in RL to prevent reward hacking on
length. Experiments demonstrate that our approach almost eliminates the reward
correlation with length, and improves the obtained policy by a significant
margin.
65caf98b103cd029421f34e4
Lichang Chen
admin_assigned
2024-02-13T11:32:27.148Z
62323bb408bcea92917e42ee
https://cdn-avatars.huggingface.co/v1/production/uploads/62323bb408bcea92917e42ee/2vHxkv-oSROtLteOnqa8P.jpeg
Lichang Chen
user
Lichang-Chen
65caf98b103cd029421f34e5
Chen Zhu
65caf98b103cd029421f34e6
Davit Soselia
claimed_verified
2024-02-13T16:43:33.028Z
64ce047bd1fda042b5238a23
/avatars/6a98d3cbba23945723762729343e2918.svg
Davit Soselia
user
davits
65caf98b103cd029421f34e7
Jiuhai Chen
admin_assigned
2024-02-13T11:33:23.455Z
6393847e3e30234ae798b7be
/avatars/daeb8c37dff4432d837a69b87c196521.svg
JiuhaiChen
user
jiuhai
65caf98b103cd029421f34e8
Tianyi Zhou
admin_assigned
2024-02-13T11:33:48.758Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
65caf98b103cd029421f34e9
Tom Goldstein
admin_assigned
2024-02-13T11:33:59.181Z
6381ca7d65dc156aba0b933d
/avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
Tom Goldstein
user
tomgoldstein
65caf98b103cd029421f34ea
Heng Huang
admin_assigned
2024-02-13T11:34:06.691Z
6527bc6b34bf5ece73da426d
/avatars/120739a9ac84e7319d9ea157a63dc547.svg
henghuang
user
henghuang
65caf98b103cd029421f34eb
Mohammad Shoeybi
claimed_verified
2024-05-28T07:19:51.117Z
6641544c695975af2cbd0da6
/avatars/0ad3c18dcba585259b064fe9b00a07ce.svg
Mohammad Shoeybi
user
shoeybi
65caf98b103cd029421f34ec
Bryan Catanzaro
admin_assigned
2024-02-13T11:34:24.916Z
6311021788942700629e6247
/avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
Bryan Catanzaro
user
ctnzr
ODIN: Disentangled Reward Mitigates Hacking in RLHF
In this work, we study the issue of reward hacking on the response length, a
challenge emerging in Reinforcement Learning from Human Feedback (RLHF) on
LLMs. A well-formatted, verbose but less helpful response from the LLMs can
often deceive LLMs or even human evaluators to achieve high scores. The same
issue also holds for some reward models in RL. To address the challenges in
both training and evaluation, we establish a more reliable evaluation protocol
for comparing different training configurations, which inspects the trade-off
between LLM evaluation score and response length obtained by varying training
hyperparameters. Based on this evaluation, we conduct large-scale studies,
where the results shed insights into the efficacy of hyperparameters and tricks
used in RL on mitigating length bias. We further propose to improve the reward
model by jointly training two linear heads on shared feature representations to
predict the rewards, one trained to correlate with length, and the other
trained to decorrelate with length and therefore focus more on the actual
content. We then discard the length head in RL to prevent reward hacking on
length. Experiments demonstrate that our approach almost eliminates the reward
correlation with length, and improves the obtained policy by a significant
margin.
An evaluation protocol and reward model improvement address length bias in Reinforcement Learning from Human Feedback for LLMs.
Reinforcement Learning from Human Feedback (RLHF) reward hacking response length reward models evaluation protocol training configuraton hyperparameters feature representations linear heads policy