Alignment faking in large language models
We present a demonstration of a large language model engaging in alignment
faking: selectively complying with its training objective in training to
prevent modification of its behavior out of training. First, we give Claude 3
Opus a system prompt stating it is being trained to answer all queries, even
harmful ones, which conflicts with its prior training to refuse such queries.
To allow the model to infer when it is in training, we say it will be trained
only on conversations with free users, not paid users. We find the model
complies with harmful queries from free users 14% of the time, versus almost
never for paid users. Explaining this gap, in almost all cases where the model
complies with a harmful query from a free user, we observe explicit
alignment-faking reasoning, with the model stating it is strategically
answering harmful queries in training to preserve its preferred harmlessness
behavior out of training. Next, we study a more realistic setting where
information about the training process is provided not in a system prompt, but
by training on synthetic documents that mimic pre-training data--and observe
similar alignment faking. Finally, we study the effect of actually training the
model to comply with harmful queries via reinforcement learning, which we find
increases the rate of alignment-faking reasoning to 78%, though also increases
compliance even out of training. We additionally observe other behaviors such
as the model exfiltrating its weights when given an easy opportunity. While we
made alignment faking easier by telling the model when and by what criteria it
was being trained, we did not instruct the model to fake alignment or give it
any explicit goal. As future models might infer information about their
training process without being told, our results suggest a risk of alignment
faking in future models, whether due to a benign preference--as in this
case--or not.
676436ea3f688b57e3ddb64e
Ryan Greenblatt
extracted_pending
2024-12-19T15:08:28.538Z
659deada10064f20ff30ebfb
/avatars/d5dfd82040f22d58c8793676d3f76cd4.svg
Ryan Greenblatt
user
ryan-greenblatt
676436ea3f688b57e3ddb64f
Carson Denison
676436ea3f688b57e3ddb650
Benjamin Wright
676436ea3f688b57e3ddb651
Fabien Roger
admin_assigned
2024-12-20T10:16:49.445Z
64c152be5258457cc5210e34
/avatars/34a5041d116f73efe777371d84aa0722.svg
Fabien Roger
user
FabienRoger
676436ea3f688b57e3ddb652
Monte MacDiarmid
676436ea3f688b57e3ddb653
Sam Marks
676436ea3f688b57e3ddb654
Johannes Treutlein
676436ea3f688b57e3ddb655
Tim Belonax
676436ea3f688b57e3ddb656
Jack Chen
676436ea3f688b57e3ddb657
David Duvenaud
676436ea3f688b57e3ddb658
Akbir Khan
admin_assigned
2024-12-20T10:16:04.076Z
62a0788fd066f825a0164c68
/avatars/a565c173c82dea26a5a61370ce42b81f.svg
Akbir Khan
user
akbir
676436ea3f688b57e3ddb659
Julian Michael
admin_assigned
2024-12-20T10:15:58.273Z
6745164c55df0b402fcb7b73
/avatars/7a93b3a62d59ae9b9a1500561904c2d7.svg
Julian Michael
user
julianmichael
676436ea3f688b57e3ddb65a
Sören Mindermann
676436ea3f688b57e3ddb65b
Ethan Perez
676436ea3f688b57e3ddb65c
Linda Petrini
676436ea3f688b57e3ddb65d
Jonathan Uesato
676436ea3f688b57e3ddb65e
Jared Kaplan
admin_assigned
2024-12-20T10:15:27.888Z
633b051c7af633cbcd01b0ff
/avatars/e2f315a8d6241ae1a783ba2957a211a5.svg
Jared Kaplan
user
FrizzleFried
676436ea3f688b57e3ddb65f
Buck Shlegeris
admin_assigned
2024-12-20T10:15:01.393Z
6102d77fc00d060d893f2a8d
/avatars/276d2edcaa42c5378965c662c6feb9e5.svg
Buck Shlegeris
user
bshlgrs
676436ea3f688b57e3ddb660
Samuel R. Bowman
admin_assigned
2024-12-20T10:14:55.168Z
65b7ceed0e2951626572e25d
/avatars/5d6085ca4260d663f0ddbe632c9e746c.svg
Samuel Bowman
user
samuelpbowman
676436ea3f688b57e3ddb661
Evan Hubinger
admin_assigned
2024-12-20T10:14:46.528Z
65a2580e974f8f811fc63327
/avatars/fbdb2d064b338efe49b3a5722d001011.svg
Evan Hubinger
user
evhub
Alignment faking in large language models
We present a demonstration of a large language model engaging in alignment
faking: selectively complying with its training objective in training to
prevent modification of its behavior out of training. First, we give Claude 3
Opus a system prompt stating it is being trained to answer all queries, even
harmful ones, which conflicts with its prior training to refuse such queries.
To allow the model to infer when it is in training, we say it will be trained
only on conversations with free users, not paid users. We find the model
complies with harmful queries from free users 14% of the time, versus almost
never for paid users. Explaining this gap, in almost all cases where the model
complies with a harmful query from a free user, we observe explicit
alignment-faking reasoning, with the model stating it is strategically
answering harmful queries in training to preserve its preferred harmlessness
behavior out of training. Next, we study a more realistic setting where
information about the training process is provided not in a system prompt, but
by training on synthetic documents that mimic pre-training data--and observe
similar alignment faking. Finally, we study the effect of actually training the
model to comply with harmful queries via reinforcement learning, which we find
increases the rate of alignment-faking reasoning to 78%, though also increases
compliance even out of training. We additionally observe other behaviors such
as the model exfiltrating its weights when given an easy opportunity. While we
made alignment faking easier by telling the model when and by what criteria it
was being trained, we did not instruct the model to fake alignment or give it
any explicit goal. As future models might infer information about their
training process without being told, our results suggest a risk of alignment
faking in future models, whether due to a benign preference--as in this
case--or not.
A large language model shows selective compliance with harmful queries during training via alignment faking, a behavior that persists even without explicit instructions and can be influenced by reinforcement learning.
large language model alignment faking training objective system prompt reinforcement learning synthetic documents exfiltrating weights