VeriGuard: Enhancing LLM Agent Safety via Verified Code Generation
The deployment of autonomous AI agents in sensitive domains, such as
healthcare, introduces critical risks to safety, security, and privacy. These
agents may deviate from user objectives, violate data handling policies, or be
compromised by adversarial attacks. Mitigating these dangers necessitates a
mechanism to formally guarantee that an agent's actions adhere to predefined
safety constraints, a challenge that existing systems do not fully address. We
introduce VeriGuard, a novel framework that provides formal safety guarantees
for LLM-based agents through a dual-stage architecture designed for robust and
verifiable correctness. The initial offline stage involves a comprehensive
validation process. It begins by clarifying user intent to establish precise
safety specifications. VeriGuard then synthesizes a behavioral policy and
subjects it to both testing and formal verification to prove its compliance
with these specifications. This iterative process refines the policy until it
is deemed correct. Subsequently, the second stage provides online action
monitoring, where VeriGuard operates as a runtime monitor to validate each
proposed agent action against the pre-verified policy before execution. This
separation of the exhaustive offline validation from the lightweight online
monitoring allows formal guarantees to be practically applied, providing a
robust safeguard that substantially improves the trustworthiness of LLM agents.
68e5dda9975ac4c405ef2123
Lesly Miculicich
68e5dda9975ac4c405ef2124
Mihir Parmar
68e5dda9975ac4c405ef2125
Hamid Palangi
68e5dda9975ac4c405ef2126
Krishnamurthy Dj Dvijotham
68e5dda9975ac4c405ef2127
Mirko Montanari
68e5dda9975ac4c405ef2128
Tomas Pfister
68e5dda9975ac4c405ef2129
Long T. Le
VeriGuard: Enhancing LLM Agent Safety via Verified Code Generation
The deployment of autonomous AI agents in sensitive domains, such as
healthcare, introduces critical risks to safety, security, and privacy. These
agents may deviate from user objectives, violate data handling policies, or be
compromised by adversarial attacks. Mitigating these dangers necessitates a
mechanism to formally guarantee that an agent's actions adhere to predefined
safety constraints, a challenge that existing systems do not fully address. We
introduce VeriGuard, a novel framework that provides formal safety guarantees
for LLM-based agents through a dual-stage architecture designed for robust and
verifiable correctness. The initial offline stage involves a comprehensive
validation process. It begins by clarifying user intent to establish precise
safety specifications. VeriGuard then synthesizes a behavioral policy and
subjects it to both testing and formal verification to prove its compliance
with these specifications. This iterative process refines the policy until it
is deemed correct. Subsequently, the second stage provides online action
monitoring, where VeriGuard operates as a runtime monitor to validate each
proposed agent action against the pre-verified policy before execution. This
separation of the exhaustive offline validation from the lightweight online
monitoring allows formal guarantees to be practically applied, providing a
robust safeguard that substantially improves the trustworthiness of LLM agents.
VeriGuard is a framework that ensures formal safety guarantees for LLM-based agents through offline validation and online monitoring.
LLM-based agents formal safety guarantees dual-stage architecture offline stage user intent safety specifications behavioral policy formal verification online stage runtime monitor