The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses
  Against Llm Jailbreaks and Prompt Injections
How should we evaluate the robustness of language model defenses? Current
defenses against jailbreaks and prompt injections (which aim to prevent an
attacker from eliciting harmful knowledge or remotely triggering malicious
actions, respectively) are typically evaluated either against a static set of
harmful attack strings, or against computationally weak optimization methods
that were not designed with the defense in mind. We argue that this evaluation
process is flawed.
  Instead, we should evaluate defenses against adaptive attackers who
explicitly modify their attack strategy to counter a defense's design while
spending considerable resources to optimize their objective. By systematically
tuning and scaling general optimization techniques-gradient descent,
reinforcement learning, random search, and human-guided exploration-we bypass
12 recent defenses (based on a diverse set of techniques) with attack success
rate above 90% for most; importantly, the majority of defenses originally
reported near-zero attack success rates. We believe that future defense work
must consider stronger attacks, such as the ones we describe, in order to make
reliable and convincing claims of robustness.
68ee80fc7b0ac1ee3e5998bb
Milad Nasr
68ee80fc7b0ac1ee3e5998bc
Nicholas Carlini
68ee80fc7b0ac1ee3e5998bd
Chawin Sitawarin
68ee80fc7b0ac1ee3e5998be
Sander V. Schulhoff
claimed_verified
2025-10-15T03:13:41.208Z
632a7ea2cf7d40df9b416bb5
https://cdn-avatars.huggingface.co/v1/production/uploads/632a7ea2cf7d40df9b416bb5/uEEBjZB8Yf_nqaGYfPuLe.jpeg
Sander Schulhoff
user
Trigaten
68ee80fc7b0ac1ee3e5998bf
Jamie Hayes
68ee80fc7b0ac1ee3e5998c0
Michael Ilie
68ee80fc7b0ac1ee3e5998c1
Juliette Pluto
68ee80fc7b0ac1ee3e5998c2
Shuang Song
68ee80fc7b0ac1ee3e5998c3
Harsh Chaudhari
68ee80fc7b0ac1ee3e5998c4
Ilia Shumailov
68ee80fc7b0ac1ee3e5998c5
Abhradeep Thakurta
68ee80fc7b0ac1ee3e5998c6
Kai Yuanqing Xiao
68ee80fc7b0ac1ee3e5998c7
Andreas Terzis
68ee80fc7b0ac1ee3e5998c8
Florian Tramèr
The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses
  Against Llm Jailbreaks and Prompt Injections
How should we evaluate the robustness of language model defenses? Current
defenses against jailbreaks and prompt injections (which aim to prevent an
attacker from eliciting harmful knowledge or remotely triggering malicious
actions, respectively) are typically evaluated either against a static set of
harmful attack strings, or against computationally weak optimization methods
that were not designed with the defense in mind. We argue that this evaluation
process is flawed.
  Instead, we should evaluate defenses against adaptive attackers who
explicitly modify their attack strategy to counter a defense's design while
spending considerable resources to optimize their objective. By systematically
tuning and scaling general optimization techniques-gradient descent,
reinforcement learning, random search, and human-guided exploration-we bypass
12 recent defenses (based on a diverse set of techniques) with attack success
rate above 90% for most; importantly, the majority of defenses originally
reported near-zero attack success rates. We believe that future defense work
must consider stronger attacks, such as the ones we describe, in order to make
reliable and convincing claims of robustness.
Defenses against jailbreaks and prompt injections in language models should be evaluated against adaptive attackers using advanced optimization techniques to ensure robustness.
gradient descent reinforcement learning random search human-guided exploration