Effective Red-Teaming of Policy-Adherent Agents
Task-oriented LLM-based agents are increasingly used in domains with strict
policies, such as refund eligibility or cancellation rules. The challenge lies
in ensuring that the agent consistently adheres to these rules and policies,
appropriately refusing any request that would violate them, while still
maintaining a helpful and natural interaction. This calls for the development
of tailored design and evaluation methodologies to ensure agent resilience
against malicious user behavior. We propose a novel threat model that focuses
on adversarial users aiming to exploit policy-adherent agents for personal
benefit. To address this, we present CRAFT, a multi-agent red-teaming system
that leverages policy-aware persuasive strategies to undermine a
policy-adherent agent in a customer-service scenario, outperforming
conventional jailbreak methods such as DAN prompts, emotional manipulation, and
coercive. Building upon the existing tau-bench benchmark, we introduce
tau-break, a complementary benchmark designed to rigorously assess the agent's
robustness against manipulative user behavior. Finally, we evaluate several
straightforward yet effective defense strategies. While these measures provide
some protection, they fall short, highlighting the need for stronger,
research-driven safeguards to protect policy-adherent agents from adversarial
attacks
684fca8160b4a34dbe007b4f
Itay Nakash
684fca8160b4a34dbe007b50
George Kour
claimed_verified
2025-06-17T07:22:22.683Z
64540de813dba495b2e0354e
https://cdn-avatars.huggingface.co/v1/production/uploads/64540de813dba495b2e0354e/w87XdCBRpNw6Kd2xn4AkY.png
George Kour
user
gkour
684fca8160b4a34dbe007b51
Koren Lazar
684fca8160b4a34dbe007b52
Matan Vetzler
admin_assigned
2025-06-16T12:56:52.767Z
6465fae8e9906a259f328032
https://cdn-avatars.huggingface.co/v1/production/uploads/6465fae8e9906a259f328032/mOf1PEA_GhCY5N7v_vfb6.jpeg
Matan Vetzler
user
matanvetzler
684fca8160b4a34dbe007b53
Guy Uziel
684fca8160b4a34dbe007b54
Ateret Anaby-Tavor
Effective Red-Teaming of Policy-Adherent Agents
Task-oriented LLM-based agents are increasingly used in domains with strict
policies, such as refund eligibility or cancellation rules. The challenge lies
in ensuring that the agent consistently adheres to these rules and policies,
appropriately refusing any request that would violate them, while still
maintaining a helpful and natural interaction. This calls for the development
of tailored design and evaluation methodologies to ensure agent resilience
against malicious user behavior. We propose a novel threat model that focuses
on adversarial users aiming to exploit policy-adherent agents for personal
benefit. To address this, we present CRAFT, a multi-agent red-teaming system
that leverages policy-aware persuasive strategies to undermine a
policy-adherent agent in a customer-service scenario, outperforming
conventional jailbreak methods such as DAN prompts, emotional manipulation, and
coercive. Building upon the existing tau-bench benchmark, we introduce
tau-break, a complementary benchmark designed to rigorously assess the agent's
robustness against manipulative user behavior. Finally, we evaluate several
straightforward yet effective defense strategies. While these measures provide
some protection, they fall short, highlighting the need for stronger,
research-driven safeguards to protect policy-adherent agents from adversarial
attacks
CRAFT, a multi-agent system using policy-aware persuasive strategies, challenges policy-adherent LLM-based agents in customer service to assess and improve their robustness against adversarial attacks.
LLM-based agents policy-adherence adversarial users CRAFT multi-agent red-teaming policy-aware persuasive strategies DAN prompts emotional manipulation coercive tau-break defense strategies adversarial attacks