WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially)
  Safer Language Models
We introduce WildTeaming, an automatic LLM safety red-teaming framework that
mines in-the-wild user-chatbot interactions to discover 5.7K unique clusters of
novel jailbreak tactics, and then composes multiple tactics for systematic
exploration of novel jailbreaks. Compared to prior work that performed
red-teaming via recruited human workers, gradient-based optimization, or
iterative revision with LLMs, our work investigates jailbreaks from chatbot
users who were not specifically instructed to break the system. WildTeaming
reveals previously unidentified vulnerabilities of frontier LLMs, resulting in
up to 4.6x more diverse and successful adversarial attacks compared to
state-of-the-art jailbreak methods.
  While many datasets exist for jailbreak evaluation, very few open-source
datasets exist for jailbreak training, as safety training data has been closed
even when model weights are open. With WildTeaming we create WildJailbreak, a
large-scale open-source synthetic safety dataset with 262K vanilla (direct
request) and adversarial (complex jailbreak) prompt-response pairs. To mitigate
exaggerated safety behaviors, WildJailbreak provides two contrastive types of
queries: 1) harmful queries (vanilla & adversarial) and 2) benign queries that
resemble harmful queries in form but contain no harm. As WildJailbreak
considerably upgrades the quality and scale of existing safety resources, it
uniquely enables us to examine the scaling effects of data and the interplay of
data properties and model capabilities during safety training. Through
extensive experiments, we identify the training properties that enable an ideal
balance of safety behaviors: appropriate safeguarding without over-refusal,
effective handling of vanilla and adversarial queries, and minimal, if any,
decrease in general capabilities. All components of WildJailbeak contribute to
achieving balanced safety behaviors of models.
667d935aaf1461e67e477798
Liwei Jiang
admin_assigned
2024-06-27T18:56:52.798Z
5efe5c89ff69163f6f59e628
/avatars/c7ea475c50fcce528c9b748f279058eb.svg
Liwei Jiang
user
liweijiang
667d935aaf1461e67e477799
Kavel Rao
admin_assigned
2024-06-27T18:57:00.770Z
6530cf185da070af77b69484
/avatars/44836b07fd5f9a0be3348e01267152a0.svg
Kavel Rao
user
kavelrao
667d935aaf1461e67e47779a
Seungju Han
admin_assigned
2024-06-27T18:57:11.745Z
614c569f9691b6ffd13ff2ca
/avatars/49601b76b8a13136fe140161e455b7c7.svg
Seungju Han
user
Seungjuhan
667d935aaf1461e67e47779b
Allyson Ettinger
admin_assigned
2024-06-27T18:57:18.330Z
6500b18c9d5687f825518282
/avatars/2abc8340eae436253b8b835eb8cfa505.svg
Allyson Ettinger
user
aettinger
667d935aaf1461e67e47779c
Faeze Brahman
admin_assigned
2024-06-27T18:57:24.452Z
65282b8d578679aac7888aec
https://cdn-avatars.huggingface.co/v1/production/uploads/65282b8d578679aac7888aec/dibBkhH-z1c70mJZZxJ7u.jpeg
Faeze Brahman
user
faezeb
667d935aaf1461e67e47779d
Sachin Kumar
admin_assigned
2024-06-27T18:57:35.335Z
63f24d2d7ddf724fbcc0ea9c
/avatars/3e24c1aa9c1b4066d2dd56aeb4b0f62e.svg
sachin kumar
user
sachinkumar
667d935aaf1461e67e47779e
Niloofar Mireshghallah
admin_assigned
2024-06-27T18:57:50.130Z
642154542cc2b3c39e7f47e2
/avatars/78fa026109ff7cc96726e6e3cd581587.svg
Fatemeh  Mireshghallah
user
niloofarm
667d935aaf1461e67e47779f
Ximing Lu
admin_assigned
2024-06-27T18:57:57.608Z
640928bd3461c51cf7378707
/avatars/b29fcb7388b81f8686086352f6321d06.svg
Ximing Lu
user
Ximing
667d935aaf1461e67e4777a0
Maarten Sap
admin_assigned
2024-06-27T18:58:02.732Z
62e0657168d80187d20e7b0e
/avatars/d8de3ec3edc381b34e6f908eeeea86ee.svg
Maarten Sap
user
maartensap
667d935aaf1461e67e4777a1
Yejin Choi
admin_assigned
2024-06-27T18:58:11.100Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
667d935aaf1461e67e4777a2
Nouha Dziri
extracted_pending
2024-06-27T16:29:15.580Z
653152a67aa28f32b0397d38
/avatars/2f0fc0abe38bc8cf996f9c8c7e90c46f.svg
Nouha Dziri
user
nouhad
WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially)
  Safer Language Models
We introduce WildTeaming, an automatic LLM safety red-teaming framework that
mines in-the-wild user-chatbot interactions to discover 5.7K unique clusters of
novel jailbreak tactics, and then composes multiple tactics for systematic
exploration of novel jailbreaks. Compared to prior work that performed
red-teaming via recruited human workers, gradient-based optimization, or
iterative revision with LLMs, our work investigates jailbreaks from chatbot
users who were not specifically instructed to break the system. WildTeaming
reveals previously unidentified vulnerabilities of frontier LLMs, resulting in
up to 4.6x more diverse and successful adversarial attacks compared to
state-of-the-art jailbreak methods.
  While many datasets exist for jailbreak evaluation, very few open-source
datasets exist for jailbreak training, as safety training data has been closed
even when model weights are open. With WildTeaming we create WildJailbreak, a
large-scale open-source synthetic safety dataset with 262K vanilla (direct
request) and adversarial (complex jailbreak) prompt-response pairs. To mitigate
exaggerated safety behaviors, WildJailbreak provides two contrastive types of
queries: 1) harmful queries (vanilla & adversarial) and 2) benign queries that
resemble harmful queries in form but contain no harm. As WildJailbreak
considerably upgrades the quality and scale of existing safety resources, it
uniquely enables us to examine the scaling effects of data and the interplay of
data properties and model capabilities during safety training. Through
extensive experiments, we identify the training properties that enable an ideal
balance of safety behaviors: appropriate safeguarding without over-refusal,
effective handling of vanilla and adversarial queries, and minimal, if any,
decrease in general capabilities. All components of WildJailbeak contribute to
achieving balanced safety behaviors of models.
WildTeaming discovers novel jailbreak tactics from user-chatbot interactions, creating a comprehensive open-source dataset that improves the safety and efficacy of LLM training.
LLM red-teaming jailbreak tactics adversarial attacks safety training WildJailbreak synthesize safety dataset contrastive queries safety behaviors balanced training properties