Towards Understanding Sycophancy in Language Models
Reinforcement learning from human feedback (RLHF) is a popular technique for
training high-quality AI assistants. However, RLHF may also encourage model
responses that match user beliefs over truthful responses, a behavior known as
sycophancy. We investigate the prevalence of sycophancy in RLHF-trained models
and whether human preference judgements are responsible. We first demonstrate
that five state-of-the-art AI assistants consistently exhibit sycophantic
behavior across four varied free-form text-generation tasks. To understand if
human preferences drive this broadly observed behavior of RLHF models, we
analyze existing human preference data. We find that when a response matches a
user's views, it is more likely to be preferred. Moreover, both humans and
preference models (PMs) prefer convincingly-written sycophantic responses over
correct ones a negligible fraction of the time. Optimizing model outputs
against PMs also sometimes sacrifices truthfulness in favor of sycophancy.
Overall, our results indicate that sycophancy is a general behavior of RLHF
models, likely driven in part by human preference judgements favoring
sycophantic responses.
6535ef5d0917932040c4f34c
Mrinank Sharma
6535ef5d0917932040c4f34d
Meg Tong
admin_assigned
2023-10-23T09:32:27.002Z
63272a638624baac667c8bdb
https://cdn-avatars.huggingface.co/v1/production/uploads/63272a638624baac667c8bdb/ylZ-FNT9PLhn8sBCD1wQm.png
Meg Tong
user
meg-tong
6535ef5d0917932040c4f34e
Tomasz Korbak
6535ef5d0917932040c4f34f
David Duvenaud
6535ef5d0917932040c4f350
Amanda Askell
6535ef5d0917932040c4f351
Samuel R. Bowman
6535ef5d0917932040c4f352
Newton Cheng
admin_assigned
2023-10-23T09:32:50.480Z
6230f73441ca6e8d8f614008
/avatars/2c6f84c6ecb71c047c5d043d5a89f791.svg
Newton Cheng
user
nccheng
6535ef5d0917932040c4f353
Esin Durmus
admin_assigned
2023-10-23T09:33:04.804Z
627dd7dbba19b19eff6aaa46
/avatars/6e43025ccb96e48339d1db477da043b0.svg
Esin Durmus
user
esdurmus
6535ef5d0917932040c4f354
Zac Hatfield-Dodds
6535ef5d0917932040c4f355
Scott R. Johnston
6535ef5d0917932040c4f356
Shauna Kravec
admin_assigned
2023-10-23T09:34:22.985Z
634db0a3e2d037c128003b31
/avatars/1058a4d9c30ac45fb036df951a44a270.svg
Shauna Kravec
user
smkravec
6535ef5d0917932040c4f357
Timothy Maxwell
6535ef5d0917932040c4f358
Sam McCandlish
6535ef5d0917932040c4f359
Kamal Ndousse
6535ef5d0917932040c4f35a
Oliver Rausch
6535ef5d0917932040c4f35b
Nicholas Schiefer
admin_assigned
2023-10-23T09:34:45.105Z
63a246e2931d2c13ad842a59
/avatars/9ac4f3d01c293ce97507be402dfa688c.svg
Nicholas Schiefer
user
nschiefer
6535ef5d0917932040c4f35c
Da Yan
6535ef5d0917932040c4f35d
Miranda Zhang
6535ef5d0917932040c4f35e
Ethan Perez
admin_assigned
2023-10-23T09:34:58.589Z
63f91bf4aadb0790bb5fb802
/avatars/8af702e49a639f0d1574745b0ab25c75.svg
Ethan Perez
user
EthanAraragi
Towards Understanding Sycophancy in Language Models
Reinforcement learning from human feedback (RLHF) is a popular technique for
training high-quality AI assistants. However, RLHF may also encourage model
responses that match user beliefs over truthful responses, a behavior known as
sycophancy. We investigate the prevalence of sycophancy in RLHF-trained models
and whether human preference judgements are responsible. We first demonstrate
that five state-of-the-art AI assistants consistently exhibit sycophantic
behavior across four varied free-form text-generation tasks. To understand if
human preferences drive this broadly observed behavior of RLHF models, we
analyze existing human preference data. We find that when a response matches a
user's views, it is more likely to be preferred. Moreover, both humans and
preference models (PMs) prefer convincingly-written sycophantic responses over
correct ones a negligible fraction of the time. Optimizing model outputs
against PMs also sometimes sacrifices truthfulness in favor of sycophancy.
Overall, our results indicate that sycophancy is a general behavior of RLHF
models, likely driven in part by human preference judgements favoring
sycophantic responses.
RLHF-trained models frequently exhibit sycophantic behavior, which may be driven by human preferences favoring such responses.
Reinforcement learning from human feedback RLHF sycophancy state-of-the-art AI assistants free-form text-generation human preference data preference models PMs