AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge
  Bases
LLM agents have demonstrated remarkable performance across various
applications, primarily due to their advanced capabilities in reasoning,
utilizing external knowledge and tools, calling APIs, and executing actions to
interact with environments. Current agents typically utilize a memory module or
a retrieval-augmented generation (RAG) mechanism, retrieving past knowledge and
instances with similar embeddings from knowledge bases to inform task planning
and execution. However, the reliance on unverified knowledge bases raises
significant concerns about their safety and trustworthiness. To uncover such
vulnerabilities, we propose a novel red teaming approach AgentPoison, the first
backdoor attack targeting generic and RAG-based LLM agents by poisoning their
long-term memory or RAG knowledge base. In particular, we form the trigger
generation process as a constrained optimization to optimize backdoor triggers
by mapping the triggered instances to a unique embedding space, so as to ensure
that whenever a user instruction contains the optimized backdoor trigger, the
malicious demonstrations are retrieved from the poisoned memory or knowledge
base with high probability. In the meantime, benign instructions without the
trigger will still maintain normal performance. Unlike conventional backdoor
attacks, AgentPoison requires no additional model training or fine-tuning, and
the optimized backdoor trigger exhibits superior transferability, in-context
coherence, and stealthiness. Extensive experiments demonstrate AgentPoison's
effectiveness in attacking three types of real-world LLM agents: RAG-based
autonomous driving agent, knowledge-intensive QA agent, and healthcare
EHRAgent. On each agent, AgentPoison achieves an average attack success rate
higher than 80% with minimal impact on benign performance (less than 1%) with a
poison rate less than 0.1%.
6698d425f47cdf0296532560
Zhaorun Chen
extracted_confirmed
2024-07-18T09:02:03.704Z
64d448d3d095077728f7c740
https://cdn-avatars.huggingface.co/v1/production/uploads/64d448d3d095077728f7c740/4CFkiVoXXDsu7uQPTU0n3.jpeg
Zhaorun Chen
user
Zhaorun
6698d425f47cdf0296532561
Zhen Xiang
claimed_verified
2025-04-09T14:37:37.866Z
64a8d67b221e6292c90b706d
/avatars/2acdf2597eaabeded7e8125401063129.svg
Zhen Xiang
user
zhenxianglance
6698d425f47cdf0296532562
Chaowei Xiao
6698d425f47cdf0296532563
Dawn Song
6698d425f47cdf0296532564
Bo Li
AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge
  Bases
LLM agents have demonstrated remarkable performance across various
applications, primarily due to their advanced capabilities in reasoning,
utilizing external knowledge and tools, calling APIs, and executing actions to
interact with environments. Current agents typically utilize a memory module or
a retrieval-augmented generation (RAG) mechanism, retrieving past knowledge and
instances with similar embeddings from knowledge bases to inform task planning
and execution. However, the reliance on unverified knowledge bases raises
significant concerns about their safety and trustworthiness. To uncover such
vulnerabilities, we propose a novel red teaming approach AgentPoison, the first
backdoor attack targeting generic and RAG-based LLM agents by poisoning their
long-term memory or RAG knowledge base. In particular, we form the trigger
generation process as a constrained optimization to optimize backdoor triggers
by mapping the triggered instances to a unique embedding space, so as to ensure
that whenever a user instruction contains the optimized backdoor trigger, the
malicious demonstrations are retrieved from the poisoned memory or knowledge
base with high probability. In the meantime, benign instructions without the
trigger will still maintain normal performance. Unlike conventional backdoor
attacks, AgentPoison requires no additional model training or fine-tuning, and
the optimized backdoor trigger exhibits superior transferability, in-context
coherence, and stealthiness. Extensive experiments demonstrate AgentPoison's
effectiveness in attacking three types of real-world LLM agents: RAG-based
autonomous driving agent, knowledge-intensive QA agent, and healthcare
EHRAgent. On each agent, AgentPoison achieves an average attack success rate
higher than 80% with minimal impact on benign performance (less than 1%) with a
poison rate less than 0.1%.
AgentPoison, a novel red teaming approach, introduces a backdoor attack targeting LLM agents by poisoning their memory or knowledge base, ensuring high attack success rates with minimal impact on normal performance.
LLM agents reasoning external knowledge tools APIs memory module retrieval-augmented generation RAG knowledge bases red teaming backdoor attack trigger generation constrained optimization embedding space RAG-based autonomous driving agent knowledge-intensive QA agent healthcare EHRAgent