Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak
  Supervision
Widely used alignment techniques, such as reinforcement learning from human
feedback (RLHF), rely on the ability of humans to supervise model behavior -
for example, to evaluate whether a model faithfully followed instructions or
generated safe outputs. However, future superhuman models will behave in
complex ways too difficult for humans to reliably evaluate; humans will only be
able to weakly supervise superhuman models. We study an analogy to this
problem: can weak model supervision elicit the full capabilities of a much
stronger model? We test this using a range of pretrained language models in the
GPT-4 family on natural language processing (NLP), chess, and reward modeling
tasks. We find that when we naively finetune strong pretrained models on labels
generated by a weak model, they consistently perform better than their weak
supervisors, a phenomenon we call weak-to-strong generalization. However, we
are still far from recovering the full capabilities of strong models with naive
finetuning alone, suggesting that techniques like RLHF may scale poorly to
superhuman models without further work. We find that simple methods can often
significantly improve weak-to-strong generalization: for example, when
finetuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence
loss, we can recover close to GPT-3.5-level performance on NLP tasks. Our
results suggest that it is feasible to make empirical progress today on a
fundamental challenge of aligning superhuman models.
657fa5f6e50ca9a69909621f
Collin Burns
657fa5f6e50ca9a699096220
Pavel Izmailov
admin_assigned
2023-12-18T09:29:21.544Z
6348d3d60672b21524f983d6
/avatars/935d93551c0bbd8a0cbfeaef4721206f.svg
Pavel Izmailov
user
izmailovpavel
657fa5f6e50ca9a699096221
Jan Hendrik Kirchner
admin_assigned
2023-12-18T09:30:04.154Z
63d1918df7f31a66a2d38140
/avatars/17fd871735fc8c4ede5d4faab9295778.svg
Jan Hendrik Kirchner
user
jkirchner
657fa5f6e50ca9a699096222
Bowen Baker
657fa5f6e50ca9a699096223
Leo Gao
657fa5f6e50ca9a699096224
Leopold Aschenbrenner
657fa5f6e50ca9a699096225
Yining Chen
657fa5f6e50ca9a699096226
Adrien Ecoffet
admin_assigned
2023-12-18T09:31:01.937Z
649fbee82bf2e21d9562b606
/avatars/92d593304a5f24875a573ee9bd725008.svg
Adrien Ecoffet
user
AdrienLE
657fa5f6e50ca9a699096227
Manas Joglekar
657fa5f6e50ca9a699096228
Jan Leike
657fa5f6e50ca9a699096229
Ilya Sutskever
657fa5f6e50ca9a69909622a
Jeff Wu
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak
  Supervision
Widely used alignment techniques, such as reinforcement learning from human
feedback (RLHF), rely on the ability of humans to supervise model behavior -
for example, to evaluate whether a model faithfully followed instructions or
generated safe outputs. However, future superhuman models will behave in
complex ways too difficult for humans to reliably evaluate; humans will only be
able to weakly supervise superhuman models. We study an analogy to this
problem: can weak model supervision elicit the full capabilities of a much
stronger model? We test this using a range of pretrained language models in the
GPT-4 family on natural language processing (NLP), chess, and reward modeling
tasks. We find that when we naively finetune strong pretrained models on labels
generated by a weak model, they consistently perform better than their weak
supervisors, a phenomenon we call weak-to-strong generalization. However, we
are still far from recovering the full capabilities of strong models with naive
finetuning alone, suggesting that techniques like RLHF may scale poorly to
superhuman models without further work. We find that simple methods can often
significantly improve weak-to-strong generalization: for example, when
finetuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence
loss, we can recover close to GPT-3.5-level performance on NLP tasks. Our
results suggest that it is feasible to make empirical progress today on a
fundamental challenge of aligning superhuman models.
Naive finetuning of strong pretrained models with weak supervision can improve performance but is insufficient to match full model capabilities, highlighting challenges for aligning superhuman models.
reinforcement learning from human feedback weak model supervision weak-to-strong generalization pretrained language models GPT-4 GPT-2 GPT-3.5 natural language processing chess reward modeling auxiliary confidence loss