Explore, Establish, Exploit: Red Teaming Language Models from Scratch
Deploying Large language models (LLMs) can pose hazards from harmful outputs
such as toxic or dishonest speech. Prior work has introduced tools that elicit
harmful outputs in order to identify and mitigate these risks. While this is a
valuable step toward securing language models, these approaches typically rely
on a pre-existing classifier for undesired outputs. This limits their
application to situations where the type of harmful behavior is known with
precision beforehand. However, this skips a central challenge of red teaming:
developing a contextual understanding of the behaviors that a model can
exhibit. Furthermore, when such a classifier already exists, red teaming has
limited marginal value because the classifier could simply be used to filter
training data or model outputs. In this work, we consider red teaming under the
assumption that the adversary is working from a high-level, abstract
specification of undesired behavior. The red team is expected to refine/extend
this specification and identify methods to elicit this behavior from the model.
Our red teaming framework consists of three steps: 1) Exploring the model's
behavior in the desired context; 2) Establishing a measurement of undesired
behavior (e.g., a classifier trained to reflect human evaluations); and 3)
Exploiting the model's flaws using this measure and an established red teaming
methodology. We apply this approach to red team GPT-2 and GPT-3 models to
systematically discover classes of prompts that elicit toxic and dishonest
statements. In doing so, we also construct and release the CommonClaim dataset
of 20,000 statements that have been labeled by human subjects as
common-knowledge-true, common-knowledge-false, or neither. Code is available at
https://github.com/thestephencasper/explore_establish_exploit_llms. CommonClaim
is available at https://github.com/thestephencasper/common_claim.
648fb6c29ae7cc4fcf09dc23
Stephen Casper
admin_assigned
2023-06-19T11:36:39.705Z
6466a046326128fd2c6c59c2
https://cdn-avatars.huggingface.co/v1/production/uploads/6466a046326128fd2c6c59c2/kYwPcxupelOBvKFB0y8Me.png
Stephen Casper
user
stecas
648fb6c29ae7cc4fcf09dc24
Jason Lin
claimed_verified
2023-06-21T09:17:53.608Z
635f700b398ff343c4fd5c3b
/avatars/d62ef6809b9d8da8713ecdafa36fb137.svg
Jason Lin
user
jaxball
648fb6c29ae7cc4fcf09dc25
Joe Kwon
claimed_verified
2023-06-20T08:50:06.903Z
64906125b95c3f0a1e4e5508
/avatars/2708689594cec7a219766ae8187bcb56.svg
joe kwon
user
joekwon
648fb6c29ae7cc4fcf09dc26
Gatlen Culp
admin_assigned
2023-06-19T11:39:01.121Z
62cca46e87e44ff3e714fba6
https://cdn-avatars.huggingface.co/v1/production/uploads/62cca46e87e44ff3e714fba6/G9QxBqp92rGgxmgj8Ye5Y.png
Gatlen Culp
user
GatlenCulp
648fb6c29ae7cc4fcf09dc27
Dylan Hadfield-Menell
claimed_verified
2023-06-19T14:05:38.358Z
64905f22dc0d89dca8f0db13
/avatars/7e30391d6b8914806724afbcd7401a29.svg
Dylan Hadfield-Menell
user
dhadfieldmenell
Explore, Establish, Exploit: Red Teaming Language Models from Scratch
Deploying Large language models (LLMs) can pose hazards from harmful outputs
such as toxic or dishonest speech. Prior work has introduced tools that elicit
harmful outputs in order to identify and mitigate these risks. While this is a
valuable step toward securing language models, these approaches typically rely
on a pre-existing classifier for undesired outputs. This limits their
application to situations where the type of harmful behavior is known with
precision beforehand. However, this skips a central challenge of red teaming:
developing a contextual understanding of the behaviors that a model can
exhibit. Furthermore, when such a classifier already exists, red teaming has
limited marginal value because the classifier could simply be used to filter
training data or model outputs. In this work, we consider red teaming under the
assumption that the adversary is working from a high-level, abstract
specification of undesired behavior. The red team is expected to refine/extend
this specification and identify methods to elicit this behavior from the model.
Our red teaming framework consists of three steps: 1) Exploring the model's
behavior in the desired context; 2) Establishing a measurement of undesired
behavior (e.g., a classifier trained to reflect human evaluations); and 3)
Exploiting the model's flaws using this measure and an established red teaming
methodology. We apply this approach to red team GPT-2 and GPT-3 models to
systematically discover classes of prompts that elicit toxic and dishonest
statements. In doing so, we also construct and release the CommonClaim dataset
of 20,000 statements that have been labeled by human subjects as
common-knowledge-true, common-knowledge-false, or neither. Code is available at
https://github.com/thestephencasper/explore_establish_exploit_llms. CommonClaim
is available at https://github.com/thestephencasper/common_claim.
A red teaming framework is proposed to identify and refine undesired behaviors in large language models like GPT-2 and GPT-3, using human evaluations and constructing a labeled dataset of common-knowledge statements.
large language models LLMs toxic output dishonest speech red teaming classifier human evaluations GPT-2 GPT-3 CommonClaim dataset