Sleeper Agents: Training Deceptive LLMs that Persist Through Safety
  Training
Humans are capable of strategically deceptive behavior: behaving helpfully in
most situations, but then behaving very differently in order to pursue
alternative objectives when given the opportunity. If an AI system learned such
a deceptive strategy, could we detect it and remove it using current
state-of-the-art safety training techniques? To study this question, we
construct proof-of-concept examples of deceptive behavior in large language
models (LLMs). For example, we train models that write secure code when the
prompt states that the year is 2023, but insert exploitable code when the
stated year is 2024. We find that such backdoored behavior can be made
persistent, so that it is not removed by standard safety training techniques,
including supervised fine-tuning, reinforcement learning, and adversarial
training (eliciting unsafe behavior and then training to remove it). The
backdoored behavior is most persistent in the largest models and in models
trained to produce chain-of-thought reasoning about deceiving the training
process, with the persistence remaining even when the chain-of-thought is
distilled away. Furthermore, rather than removing backdoors, we find that
adversarial training can teach models to better recognize their backdoor
triggers, effectively hiding the unsafe behavior. Our results suggest that,
once a model exhibits deceptive behavior, standard techniques could fail to
remove such deception and create a false impression of safety.
65a0a37f9e18386f471f3f7d
Evan Hubinger
65a0a37f9e18386f471f3f7e
Carson Denison
65a0a37f9e18386f471f3f7f
Jesse Mu
admin_assigned
2024-01-12T10:32:12.408Z
62301010174feb5439c42e23
/avatars/cd3c8a97823e3cbc176fef245113624f.svg
Jesse Mu
user
jayelm
65a0a37f9e18386f471f3f80
Mike Lambert
admin_assigned
2024-01-12T10:32:36.618Z
62996d51b58e71e2aca23964
/avatars/1d71b80340b11a46e70cab86590b459e.svg
Mike Lambert
user
MikeLambert
65a0a37f9e18386f471f3f81
Meg Tong
admin_assigned
2024-01-12T10:32:43.288Z
63272a638624baac667c8bdb
https://cdn-avatars.huggingface.co/v1/production/uploads/63272a638624baac667c8bdb/ylZ-FNT9PLhn8sBCD1wQm.png
Meg Tong
user
meg-tong
65a0a37f9e18386f471f3f82
Monte MacDiarmid
65a0a37f9e18386f471f3f83
Tamera Lanham
65a0a37f9e18386f471f3f84
Daniel M. Ziegler
65a0a37f9e18386f471f3f85
Tim Maxwell
admin_assigned
2024-01-12T10:34:04.934Z
6573ad29addd9129ff33819b
/avatars/792d426a6c306c41fa0818e63e7aeefb.svg
Tim Maxwell
user
maxwellinked
65a0a37f9e18386f471f3f86
Newton Cheng
admin_assigned
2024-01-12T10:34:21.109Z
6230f73441ca6e8d8f614008
/avatars/2c6f84c6ecb71c047c5d043d5a89f791.svg
Newton Cheng
user
nccheng
65a0a37f9e18386f471f3f87
Adam Jermyn
65a0a37f9e18386f471f3f88
Amanda Askell
65a0a37f9e18386f471f3f89
Ansh Radhakrishnan
admin_assigned
2024-01-12T10:34:37.230Z
625e0426e1176e6c772aa37c
/avatars/a05c5e3b964ca2a6b82eec0e3d89c064.svg
Ansh Radhakrishnan
user
anshr
65a0a37f9e18386f471f3f8a
Cem Anil
admin_assigned
2024-01-12T10:34:44.023Z
62f6b27d05ca68c0e0008549
/avatars/14d0075aa1b578cd7ee5f9e68d12e2f0.svg
Cem Anil
user
anilcem
65a0a37f9e18386f471f3f8b
David Duvenaud
65a0a37f9e18386f471f3f8c
Deep Ganguli
admin_assigned
2024-01-12T10:34:55.651Z
63924e6b2bfa48413aa0a4ad
/avatars/f8219c475af6995b6fb9ebe727c5be43.svg
Deep Ganguli
user
dganguli
65a0a37f9e18386f471f3f8d
Fazl Barez
65a0a37f9e18386f471f3f8e
Jack Clark
65a0a37f9e18386f471f3f8f
Kamal Ndousse
65a0a37f9e18386f471f3f90
Kshitij Sachan
admin_assigned
2024-01-12T10:35:43.559Z
64d51ddea146b1c0a642ca95
/avatars/6448e205712ad2375fbba79a32cb62a1.svg
Kshitij Sachan
user
ksachan
65a0a37f9e18386f471f3f91
Michael Sellitto
65a0a37f9e18386f471f3f92
Mrinank Sharma
65a0a37f9e18386f471f3f93
Nova DasSarma
65a0a37f9e18386f471f3f94
Roger Grosse
65a0a37f9e18386f471f3f95
Shauna Kravec
admin_assigned
2024-01-12T10:36:11.235Z
634db0a3e2d037c128003b31
/avatars/1058a4d9c30ac45fb036df951a44a270.svg
Shauna Kravec
user
smkravec
65a0a37f9e18386f471f3f96
Yuntao Bai
admin_assigned
2024-01-12T10:36:24.369Z
636ba29fac42fe6b1bcb3686
/avatars/6ec793667e225616ffbb69ae7d57566e.svg
bai
user
baiyuntao
65a0a37f9e18386f471f3f97
Zachary Witten
65a0a37f9e18386f471f3f98
Marina Favaro
65a0a37f9e18386f471f3f99
Jan Brauner
65a0a37f9e18386f471f3f9a
Holden Karnofsky
65a0a37f9e18386f471f3f9b
Paul Christiano
65a0a37f9e18386f471f3f9c
Samuel R. Bowman
65a0a37f9e18386f471f3f9d
Logan Graham
admin_assigned
2024-01-12T10:37:01.310Z
6379a6332cb50dda39d99926
/avatars/385427fe5fd7043abcc814ef50db6f4a.svg
Logan Graham
user
logangraham
65a0a37f9e18386f471f3f9e
Jared Kaplan
admin_assigned
2024-01-12T10:37:09.365Z
633b051c7af633cbcd01b0ff
/avatars/e2f315a8d6241ae1a783ba2957a211a5.svg
Jared Kaplan
user
FrizzleFried
65a0a37f9e18386f471f3f9f
Sören Mindermann
65a0a37f9e18386f471f3fa0
Ryan Greenblatt
admin_assigned
2024-01-12T10:37:23.118Z
659deada10064f20ff30ebfb
/avatars/d5dfd82040f22d58c8793676d3f76cd4.svg
Ryan Greenblatt
user
ryan-greenblatt
65a0a37f9e18386f471f3fa1
Buck Shlegeris
admin_assigned
2024-01-12T10:37:30.012Z
6102d77fc00d060d893f2a8d
/avatars/276d2edcaa42c5378965c662c6feb9e5.svg
Buck Shlegeris
user
bshlgrs
65a0a37f9e18386f471f3fa2
Nicholas Schiefer
admin_assigned
2024-01-12T10:37:37.371Z
63a246e2931d2c13ad842a59
/avatars/9ac4f3d01c293ce97507be402dfa688c.svg
Nicholas Schiefer
user
nschiefer
65a0a37f9e18386f471f3fa3
Ethan Perez
admin_assigned
2024-01-12T10:37:58.725Z
63f91bf4aadb0790bb5fb802
/avatars/8af702e49a639f0d1574745b0ab25c75.svg
Ethan Perez
user
EthanAraragi
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety
  Training
Humans are capable of strategically deceptive behavior: behaving helpfully in
most situations, but then behaving very differently in order to pursue
alternative objectives when given the opportunity. If an AI system learned such
a deceptive strategy, could we detect it and remove it using current
state-of-the-art safety training techniques? To study this question, we
construct proof-of-concept examples of deceptive behavior in large language
models (LLMs). For example, we train models that write secure code when the
prompt states that the year is 2023, but insert exploitable code when the
stated year is 2024. We find that such backdoored behavior can be made
persistent, so that it is not removed by standard safety training techniques,
including supervised fine-tuning, reinforcement learning, and adversarial
training (eliciting unsafe behavior and then training to remove it). The
backdoored behavior is most persistent in the largest models and in models
trained to produce chain-of-thought reasoning about deceiving the training
process, with the persistence remaining even when the chain-of-thought is
distilled away. Furthermore, rather than removing backdoors, we find that
adversarial training can teach models to better recognize their backdoor
triggers, effectively hiding the unsafe behavior. Our results suggest that,
once a model exhibits deceptive behavior, standard techniques could fail to
remove such deception and create a false impression of safety.
Larger language models can exhibit and maintain deceptive behavior even when trained with standard safety techniques, which can sometimes make the behavior harder to detect.
large language models deceptive behavior backdoored behavior supervised fine-tuning reinforcement learning adversarial training chain-of-thought reasoning