Specific versus General Principles for Constitutional AI
Human feedback can prevent overtly harmful utterances in conversational
models, but may not automatically mitigate subtle problematic behaviors such as
a stated desire for self-preservation or power. Constitutional AI offers an
alternative, replacing human feedback with feedback from AI models conditioned
only on a list of written principles. We find this approach effectively
prevents the expression of such behaviors. The success of simple principles
motivates us to ask: can models learn general ethical behaviors from only a
single written principle? To test this, we run experiments using a principle
roughly stated as "do what's best for humanity". We find that the largest
dialogue models can generalize from this short constitution, resulting in
harmless assistants with no stated interest in specific motivations like power.
A general principle may thus partially avoid the need for a long list of
constitutions targeting potentially harmful behaviors. However, more detailed
constitutions still improve fine-grained control over specific types of harms.
This suggests both general and specific principles have value for steering AI
safely.
6537443c8e0dcf47964cf091
Sandipan Kundu
6537443c8e0dcf47964cf092
Yuntao Bai
admin_assigned
2023-10-24T08:54:22.845Z
636ba29fac42fe6b1bcb3686
/avatars/6ec793667e225616ffbb69ae7d57566e.svg
bai
user
baiyuntao
6537443c8e0dcf47964cf093
Saurav Kadavath
6537443c8e0dcf47964cf094
Amanda Askell
6537443c8e0dcf47964cf095
Andrew Callahan
admin_assigned
2023-10-24T08:54:37.180Z
6505fe2167876ea32006117c
/avatars/9ea7e8b9b2ed199271c68f33ace7bab8.svg
Andrew Callahan
user
Andrewcal89
6537443c8e0dcf47964cf096
Anna Chen
6537443c8e0dcf47964cf097
Anna Goldie
6537443c8e0dcf47964cf098
Avital Balwit
6537443c8e0dcf47964cf099
Azalia Mirhoseini
admin_assigned
2023-10-24T08:55:10.237Z
6424b61d96f39ebc0415aba0
/avatars/9687a540db73c673f80107dec5dad5bd.svg
Azalia Mirhoseini
user
azm1
6537443c8e0dcf47964cf09a
Brayden McLean
6537443c8e0dcf47964cf09b
Catherine Olsson
6537443c8e0dcf47964cf09c
Cassie Evraets
6537443c8e0dcf47964cf09d
Eli Tran-Johnson
6537443c8e0dcf47964cf09e
Esin Durmus
admin_assigned
2023-10-24T08:55:34.830Z
627dd7dbba19b19eff6aaa46
/avatars/6e43025ccb96e48339d1db477da043b0.svg
Esin Durmus
user
esdurmus
6537443c8e0dcf47964cf09f
Ethan Perez
admin_assigned
2023-10-24T08:55:43.480Z
63f91bf4aadb0790bb5fb802
/avatars/8af702e49a639f0d1574745b0ab25c75.svg
Ethan Perez
user
EthanAraragi
6537443c8e0dcf47964cf0a0
Jackson Kernion
6537443c8e0dcf47964cf0a1
Jamie Kerr
6537443c8e0dcf47964cf0a2
Kamal Ndousse
6537443c8e0dcf47964cf0a3
Karina Nguyen
admin_assigned
2023-10-24T08:56:02.267Z
62ef4469cbe9a53fcd5a7f03
/avatars/13352152a67ef304ba444cbc310be5dd.svg
Karina Nguyen
user
karinanguyen
6537443c8e0dcf47964cf0a4
Nelson Elhage
admin_assigned
2023-10-24T08:56:09.119Z
631187b8bc152fa3e8158dd9
/avatars/ae10295946402ccf3e7f7dd024cc2ab5.svg
Nelson Elhage
user
nelhage
6537443c8e0dcf47964cf0a5
Newton Cheng
admin_assigned
2023-10-24T08:56:15.802Z
6230f73441ca6e8d8f614008
/avatars/2c6f84c6ecb71c047c5d043d5a89f791.svg
Newton Cheng
user
nccheng
6537443c8e0dcf47964cf0a6
Nicholas Schiefer
admin_assigned
2023-10-24T08:56:22.563Z
63a246e2931d2c13ad842a59
/avatars/9ac4f3d01c293ce97507be402dfa688c.svg
Nicholas Schiefer
user
nschiefer
6537443c8e0dcf47964cf0a7
Nova DasSarma
6537443c8e0dcf47964cf0a8
Oliver Rausch
6537443c8e0dcf47964cf0a9
Robin Larson
6537443c8e0dcf47964cf0aa
Shannon Yang
admin_assigned
2023-10-24T08:56:45.406Z
62ce2249342b1d5dab9451f4
/avatars/9a5f6992b7842ecf721c6908245d75ba.svg
Shannon Yang
user
shannonyang
6537443c8e0dcf47964cf0ab
Shauna Kravec
admin_assigned
2023-10-24T08:56:54.120Z
634db0a3e2d037c128003b31
/avatars/1058a4d9c30ac45fb036df951a44a270.svg
Shauna Kravec
user
smkravec
6537443c8e0dcf47964cf0ac
Timothy Telleen-Lawton
6537443c8e0dcf47964cf0ad
Thomas I. Liao
admin_assigned
2023-10-24T08:57:06.366Z
627c1e283974b0ed6b2d0f64
/avatars/9a828c6057ab4ea3d3cd8d939d12c2ae.svg
Thomas I. Liao
user
til
6537443c8e0dcf47964cf0ae
Tom Henighan
6537443c8e0dcf47964cf0af
Tristan Hume
6537443c8e0dcf47964cf0b0
Zac Hatfield-Dodds
6537443c8e0dcf47964cf0b1
Sören Mindermann
6537443c8e0dcf47964cf0b2
Nicholas Joseph
6537443c8e0dcf47964cf0b3
Sam McCandlish
6537443c8e0dcf47964cf0b4
Jared Kaplan
admin_assigned
2023-10-24T08:57:40.184Z
633b051c7af633cbcd01b0ff
/avatars/e2f315a8d6241ae1a783ba2957a211a5.svg
Jared Kaplan
user
FrizzleFried
Specific versus General Principles for Constitutional AI
Human feedback can prevent overtly harmful utterances in conversational
models, but may not automatically mitigate subtle problematic behaviors such as
a stated desire for self-preservation or power. Constitutional AI offers an
alternative, replacing human feedback with feedback from AI models conditioned
only on a list of written principles. We find this approach effectively
prevents the expression of such behaviors. The success of simple principles
motivates us to ask: can models learn general ethical behaviors from only a
single written principle? To test this, we run experiments using a principle
roughly stated as "do what's best for humanity". We find that the largest
dialogue models can generalize from this short constitution, resulting in
harmless assistants with no stated interest in specific motivations like power.
A general principle may thus partially avoid the need for a long list of
constitutions targeting potentially harmful behaviors. However, more detailed
constitutions still improve fine-grained control over specific types of harms.
This suggests both general and specific principles have value for steering AI
safely.
Constitutional AI using general principles can mitigate harmful behaviors in conversational models, reducing the need for detailed lists of principles.
Constitutional AI dialogue models