ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning
Autonomous agents powered by foundation models have seen widespread adoption
across various real-world applications. However, they remain highly vulnerable
to malicious instructions and attacks, which can result in severe consequences
such as privacy breaches and financial losses. More critically, existing
guardrails for LLMs are not applicable due to the complex and dynamic nature of
agents. To tackle these challenges, we propose ShieldAgent, the first guardrail
agent designed to enforce explicit safety policy compliance for the action
trajectory of other protected agents through logical reasoning. Specifically,
ShieldAgent first constructs a safety policy model by extracting verifiable
rules from policy documents and structuring them into a set of action-based
probabilistic rule circuits. Given the action trajectory of the protected
agent, ShieldAgent retrieves relevant rule circuits and generates a shielding
plan, leveraging its comprehensive tool library and executable code for formal
verification. In addition, given the lack of guardrail benchmarks for agents,
we introduce ShieldAgent-Bench, a dataset with 3K safety-related pairs of agent
instructions and action trajectories, collected via SOTA attacks across 6 web
environments and 7 risk categories. Experiments show that ShieldAgent achieves
SOTA on ShieldAgent-Bench and three existing benchmarks, outperforming prior
methods by 11.3% on average with a high recall of 90.1%. Additionally,
ShieldAgent reduces API queries by 64.7% and inference time by 58.2%,
demonstrating its high precision and efficiency in safeguarding agents.
67f43e4d79e20b02392d9e33
Zhaorun Chen
67f43e4d79e20b02392d9e34
Mintong Kang
67f43e4d79e20b02392d9e35
Bo Li
ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning
Autonomous agents powered by foundation models have seen widespread adoption
across various real-world applications. However, they remain highly vulnerable
to malicious instructions and attacks, which can result in severe consequences
such as privacy breaches and financial losses. More critically, existing
guardrails for LLMs are not applicable due to the complex and dynamic nature of
agents. To tackle these challenges, we propose ShieldAgent, the first guardrail
agent designed to enforce explicit safety policy compliance for the action
trajectory of other protected agents through logical reasoning. Specifically,
ShieldAgent first constructs a safety policy model by extracting verifiable
rules from policy documents and structuring them into a set of action-based
probabilistic rule circuits. Given the action trajectory of the protected
agent, ShieldAgent retrieves relevant rule circuits and generates a shielding
plan, leveraging its comprehensive tool library and executable code for formal
verification. In addition, given the lack of guardrail benchmarks for agents,
we introduce ShieldAgent-Bench, a dataset with 3K safety-related pairs of agent
instructions and action trajectories, collected via SOTA attacks across 6 web
environments and 7 risk categories. Experiments show that ShieldAgent achieves
SOTA on ShieldAgent-Bench and three existing benchmarks, outperforming prior
methods by 11.3% on average with a high recall of 90.1%. Additionally,
ShieldAgent reduces API queries by 64.7% and inference time by 58.2%,
demonstrating its high precision and efficiency in safeguarding agents.
ShieldAgent enforces safety policy compliance for autonomous agents using logical reasoning and probabilistic rule circuits, outperforming existing methods on benchmarks while reducing computational overhead.
LLMs ShieldAgent safety policy compliance action trajectory logical reasoning probabilistic rule circuits ShieldAgent-Bench API queries inference time