Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts
As large language models (LLMs) become increasingly prevalent across many
real-world applications, understanding and enhancing their robustness to user
inputs is of paramount importance. Existing methods for identifying adversarial
prompts tend to focus on specific domains, lack diversity, or require extensive
human annotations. To address these limitations, we present Rainbow Teaming, a
novel approach for producing a diverse collection of adversarial prompts.
Rainbow Teaming casts adversarial prompt generation as a quality-diversity
problem, and uses open-ended search to generate prompts that are both effective
and diverse. It can uncover a model's vulnerabilities across a broad range of
domains including, in this paper, safety, question answering, and
cybersecurity. We also demonstrate that fine-tuning on synthetic data generated
by Rainbow Teaming improves the safety of state-of-the-art LLMs without hurting
their general capabilities and helpfulness, paving the path to open-ended
self-improvement.
65dd759e89a2a760d1637c32
Mikayel Samvelyan
admin_assigned
2024-02-27T10:42:57.674Z
64f650648afa37b1248c8792
https://cdn-avatars.huggingface.co/v1/production/uploads/64f650648afa37b1248c8792/jqYcQWIdLrnBXASnVBUp-.jpeg
Mikayel Samvelyan
user
samvelyan
65dd759e89a2a760d1637c33
Sharath Chandra Raparthy
admin_assigned
2024-02-27T10:43:17.402Z
65007a96ea5699b59a9bb432
/avatars/4549f5964b007de189c45a77daf10075.svg
Sharath Raparthy
user
sharathraparthy
65dd759e89a2a760d1637c34
Andrei Lupu
admin_assigned
2024-02-27T10:43:23.914Z
64c3ffbb17bd80604501fc28
/avatars/cfe60c2c6b5656cb541357877aa757dc.svg
Andrei Lupu
user
iamexploit
65dd759e89a2a760d1637c35
Eric Hambro
65dd759e89a2a760d1637c36
Aram H. Markosyan
admin_assigned
2024-02-27T10:43:36.607Z
63fe54211193ba2c229bd74a
/avatars/14c7cc8b9ebde8d6d4f16e578e311396.svg
Aram Markosyan
user
armarkos
65dd759e89a2a760d1637c37
Manish Bhatt
65dd759e89a2a760d1637c38
Yuning Mao
admin_assigned
2024-02-27T10:44:17.726Z
62bf46a630a1888a761f8619
/avatars/de6c656066b1e88d37882ce68a0fd25c.svg
Yuning Mao
user
yuningm
65dd759e89a2a760d1637c39
Minqi Jiang
admin_assigned
2024-02-27T10:45:23.233Z
630368320547362a22a21798
/avatars/035b468584e43b4fbbf4c4aa1d2bbac7.svg
Minqi Jiang
user
minqi
65dd759e89a2a760d1637c3a
Jack Parker-Holder
65dd759e89a2a760d1637c3b
Jakob Foerster
65dd759e89a2a760d1637c3c
Tim Rocktäschel
65dd759e89a2a760d1637c3d
Roberta Raileanu
admin_assigned
2024-02-27T10:45:45.216Z
633e94793a17ab61de8e2b9c
/avatars/5f2f58ddeed211393660ada6b135f0d5.svg
Roberta Raileanu
user
rraileanu
Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts
As large language models (LLMs) become increasingly prevalent across many
real-world applications, understanding and enhancing their robustness to user
inputs is of paramount importance. Existing methods for identifying adversarial
prompts tend to focus on specific domains, lack diversity, or require extensive
human annotations. To address these limitations, we present Rainbow Teaming, a
novel approach for producing a diverse collection of adversarial prompts.
Rainbow Teaming casts adversarial prompt generation as a quality-diversity
problem, and uses open-ended search to generate prompts that are both effective
and diverse. It can uncover a model's vulnerabilities across a broad range of
domains including, in this paper, safety, question answering, and
cybersecurity. We also demonstrate that fine-tuning on synthetic data generated
by Rainbow Teaming improves the safety of state-of-the-art LLMs without hurting
their general capabilities and helpfulness, paving the path to open-ended
self-improvement.
Rainbow Teaming, an approach to generating diverse adversarial prompts, enhances the robustness of large language models across multiple domains by improving their safety without compromising general performance.
large language models (LLMs) adversarial prompts quality-diversity problem open-ended search safety question answering cybersecurity fine-tuning synthetic data open-ended self-improvement