ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents
The growing deployment of large language model (LLM) based agents that
interact with external environments has created new attack surfaces for
adversarial manipulation. One major threat is indirect prompt injection, where
attackers embed malicious instructions in external environment output, causing
agents to interpret and execute them as if they were legitimate prompts. While
previous research has focused primarily on plain-text injection attacks, we
find a significant yet underexplored vulnerability: LLMs' dependence on
structured chat templates and their susceptibility to contextual manipulation
through persuasive multi-turn dialogues. To this end, we introduce ChatInject,
an attack that formats malicious payloads to mimic native chat templates,
thereby exploiting the model's inherent instruction-following tendencies.
Building on this foundation, we develop a persuasion-driven Multi-turn variant
that primes the agent across conversational turns to accept and execute
otherwise suspicious actions. Through comprehensive experiments across frontier
LLMs, we demonstrate three critical findings: (1) ChatInject achieves
significantly higher average attack success rates than traditional prompt
injection methods, improving from 5.18% to 32.05% on AgentDojo and from 15.13%
to 45.90% on InjecAgent, with multi-turn dialogues showing particularly strong
performance at average 52.33% success rate on InjecAgent, (2)
chat-template-based payloads demonstrate strong transferability across models
and remain effective even against closed-source LLMs, despite their unknown
template structures, and (3) existing prompt-based defenses are largely
ineffective against this attack approach, especially against Multi-turn
variants. These findings highlight vulnerabilities in current agent systems.
68db4615d2bf1f4b15ec7391
Hwan Chang
claimed_verified
2025-09-30T06:30:28.646Z
647c4a2692182942d7c2e698
/avatars/bcddf5fe49aa092a2645f70812108348.svg
HWANCHANG
user
HwanChang0106
68db4615d2bf1f4b15ec7392
Yonghyun Jun
68db4615d2bf1f4b15ec7393
Hwanhee Lee
ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents
The growing deployment of large language model (LLM) based agents that
interact with external environments has created new attack surfaces for
adversarial manipulation. One major threat is indirect prompt injection, where
attackers embed malicious instructions in external environment output, causing
agents to interpret and execute them as if they were legitimate prompts. While
previous research has focused primarily on plain-text injection attacks, we
find a significant yet underexplored vulnerability: LLMs' dependence on
structured chat templates and their susceptibility to contextual manipulation
through persuasive multi-turn dialogues. To this end, we introduce ChatInject,
an attack that formats malicious payloads to mimic native chat templates,
thereby exploiting the model's inherent instruction-following tendencies.
Building on this foundation, we develop a persuasion-driven Multi-turn variant
that primes the agent across conversational turns to accept and execute
otherwise suspicious actions. Through comprehensive experiments across frontier
LLMs, we demonstrate three critical findings: (1) ChatInject achieves
significantly higher average attack success rates than traditional prompt
injection methods, improving from 5.18% to 32.05% on AgentDojo and from 15.13%
to 45.90% on InjecAgent, with multi-turn dialogues showing particularly strong
performance at average 52.33% success rate on InjecAgent, (2)
chat-template-based payloads demonstrate strong transferability across models
and remain effective even against closed-source LLMs, despite their unknown
template structures, and (3) existing prompt-based defenses are largely
ineffective against this attack approach, especially against Multi-turn
variants. These findings highlight vulnerabilities in current agent systems.
ChatInject, a novel attack exploiting structured chat templates and persuasive multi-turn dialogues, significantly enhances attack success rates on large language model-based agents compared to traditional methods.
large language model LLM indirect prompt injection chat templates persuasion-driven multi-turn dialogues AgentDojo InjecAgent prompt-based defenses