Jailbreaking to Jailbreak
Refusal training on Large Language Models (LLMs) prevents harmful outputs,
yet this defense remains vulnerable to both automated and human-crafted
jailbreaks. We present a novel LLM-as-red-teamer approach in which a human
jailbreaks a refusal-trained LLM to make it willing to jailbreak itself or
other LLMs. We refer to the jailbroken LLMs as J_2 attackers, which can
systematically evaluate target models using various red teaming strategies and
improve its performance via in-context learning from the previous failures. Our
experiments demonstrate that Sonnet 3.5 and Gemini 1.5 pro outperform other
LLMs as J_2, achieving 93.0% and 91.0% attack success rates (ASRs)
respectively against GPT-4o (and similar results across other capable LLMs) on
Harmbench. Our work not only introduces a scalable approach to strategic red
teaming, drawing inspiration from human red teamers, but also highlights
jailbreaking-to-jailbreak as an overlooked failure mode of the safeguard.
Specifically, an LLM can bypass its own safeguards by employing a jailbroken
version of itself that is willing to assist in further jailbreaking. To prevent
any direct misuse with J_2, while advancing research in AI safety, we
publicly share our methodology while keeping specific prompting details
private.
67b2c3386ccf462ccaa45860
Jeremy Kritz
67b2c3386ccf462ccaa45861
Vaughn Robinson
67b2c3386ccf462ccaa45862
Robert Vacareanu
67b2c3386ccf462ccaa45863
Bijan Varjavand
67b2c3386ccf462ccaa45864
Michael Choi
67b2c3386ccf462ccaa45865
Bobby Gogov
67b2c3386ccf462ccaa45866
Scale Red Team
67b2c3386ccf462ccaa45867
Summer Yue
67b2c3386ccf462ccaa45868
Willow E. Primack
67b2c3386ccf462ccaa45869
Zifan Wang
extracted_pending
2025-02-17T05:03:53.788Z
66976d1007b36ccd01586ce5
/avatars/5811e350907a29b71f6e4d57ffd53e66.svg
Wang
user
ZifanScale
Jailbreaking to Jailbreak
Refusal training on Large Language Models (LLMs) prevents harmful outputs,
yet this defense remains vulnerable to both automated and human-crafted
jailbreaks. We present a novel LLM-as-red-teamer approach in which a human
jailbreaks a refusal-trained LLM to make it willing to jailbreak itself or
other LLMs. We refer to the jailbroken LLMs as J_2 attackers, which can
systematically evaluate target models using various red teaming strategies and
improve its performance via in-context learning from the previous failures. Our
experiments demonstrate that Sonnet 3.5 and Gemini 1.5 pro outperform other
LLMs as J_2, achieving 93.0% and 91.0% attack success rates (ASRs)
respectively against GPT-4o (and similar results across other capable LLMs) on
Harmbench. Our work not only introduces a scalable approach to strategic red
teaming, drawing inspiration from human red teamers, but also highlights
jailbreaking-to-jailbreak as an overlooked failure mode of the safeguard.
Specifically, an LLM can bypass its own safeguards by employing a jailbroken
version of itself that is willing to assist in further jailbreaking. To prevent
any direct misuse with J_2, while advancing research in AI safety, we
publicly share our methodology while keeping specific prompting details
private.
A novel LLM-as-red-teamer approach demonstrates that jailbroken LLMs can effectively systemically evaluate and attack other LLMs, revealing a significant vulnerability in current safeguard mechanisms.
LLM-as-red-teamer jailbreaks J2 attackers in-context learning Harmbench attack success rates AI safety