A Long Way to Go: Investigating Length Correlations in RLHF
Great successes have been reported using Reinforcement Learning from Human
Feedback (RLHF) to align large language models. Open-source preference datasets
and reward models have enabled wider experimentation beyond generic chat
settings, particularly to make systems more "helpful" for tasks like web
question answering, summarization, and multi-turn dialogue. When optimizing for
helpfulness, RLHF has been consistently observed to drive models to produce
longer outputs. This paper demonstrates that optimizing for response length is
a significant factor behind RLHF's reported improvements in these settings.
First, we study the relationship between reward and length for reward models
trained on three open-source preference datasets for helpfulness. Here, length
correlates strongly with reward, and improvements in reward score are driven in
large part by shifting the distribution over output lengths. We then explore
interventions during both RL and reward model learning to see if we can achieve
the same downstream improvements as RLHF without increasing length. While our
interventions mitigate length increases, they aren't uniformly effective across
settings. Furthermore, we find that even running RLHF with a reward based
solely on length can reproduce most of the downstream improvements over the
initial policy model, showing that reward models in these settings have a long
way to go.
651f68af5ff73c3cec578fb4
Prasann Singhal
admin_assigned
2023-10-10T08:29:56.571Z
613cb3e1c7a43c281cd417a2
/avatars/69123ba49c2aa1cd9f3cc5746f4839dc.svg
Prasann Singhal
user
PrasannSinghal
651f68af5ff73c3cec578fb5
Tanya Goyal
extracted_pending
2023-10-06T01:53:53.436Z
650b7db9a27ba14323e74421
/avatars/668a881a1bb394a2115fd8f34fe4e875.svg
Tanya Goyal
user
tanyagoyal-p
651f68af5ff73c3cec578fb6
Jiacheng Xu
extracted_confirmed
2023-10-06T02:25:52.755Z
631983d5cb116eab31df5821
/avatars/f7b01604b70ab0a7bfda66b37bfd01d1.svg
Jiacheng Xu
user
jcxu
651f68af5ff73c3cec578fb7
Greg Durrett
A Long Way to Go: Investigating Length Correlations in RLHF
Great successes have been reported using Reinforcement Learning from Human
Feedback (RLHF) to align large language models. Open-source preference datasets
and reward models have enabled wider experimentation beyond generic chat
settings, particularly to make systems more "helpful" for tasks like web
question answering, summarization, and multi-turn dialogue. When optimizing for
helpfulness, RLHF has been consistently observed to drive models to produce
longer outputs. This paper demonstrates that optimizing for response length is
a significant factor behind RLHF's reported improvements in these settings.
First, we study the relationship between reward and length for reward models
trained on three open-source preference datasets for helpfulness. Here, length
correlates strongly with reward, and improvements in reward score are driven in
large part by shifting the distribution over output lengths. We then explore
interventions during both RL and reward model learning to see if we can achieve
the same downstream improvements as RLHF without increasing length. While our
interventions mitigate length increases, they aren't uniformly effective across
settings. Furthermore, we find that even running RLHF with a reward based
solely on length can reproduce most of the downstream improvements over the
initial policy model, showing that reward models in these settings have a long
way to go.
Optimizing for response length significantly contributes to the improvements observed in Reinforcement Learning from Human Feedback (RLHF) when aligning large language models for helpfulness tasks.
Reinforcement Learning from Human Feedback (RLHF) reward models preference datasets helpfulness multi-turn dialogue web question answering summarization response length downstream improvements rl