AdInject: Real-World Black-Box Attacks on Web Agents via Advertising
  Delivery
Vision-Language Model (VLM) based Web Agents represent a significant step
towards automating complex tasks by simulating human-like interaction with
websites. However, their deployment in uncontrolled web environments introduces
significant security vulnerabilities. Existing research on adversarial
environmental injection attacks often relies on unrealistic assumptions, such
as direct HTML manipulation, knowledge of user intent, or access to agent model
parameters, limiting their practical applicability. In this paper, we propose
AdInject, a novel and real-world black-box attack method that leverages the
internet advertising delivery to inject malicious content into the Web Agent's
environment. AdInject operates under a significantly more realistic threat
model than prior work, assuming a black-box agent, static malicious content
constraints, and no specific knowledge of user intent. AdInject includes
strategies for designing malicious ad content aimed at misleading agents into
clicking, and a VLM-based ad content optimization technique that infers
potential user intents from the target website's context and integrates these
intents into the ad content to make it appear more relevant or critical to the
agent's task, thus enhancing attack effectiveness. Experimental evaluations
demonstrate the effectiveness of AdInject, attack success rates exceeding 60%
in most scenarios and approaching 100% in certain cases. This strongly
demonstrates that prevalent advertising delivery constitutes a potent and
real-world vector for environment injection attacks against Web Agents. This
work highlights a critical vulnerability in Web Agent security arising from
real-world environment manipulation channels, underscoring the urgent need for
developing robust defense mechanisms against such threats. Our code is
available at https://github.com/NicerWang/AdInject.
683681c189cf92972059d4e8
Haowei Wang
claimed_verified
2025-05-28T08:57:25.748Z
63a85367353e10031a8becaa
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1Zb_T08yt0jJCHTe-ahGE.png
NicerWang
user
NicerWang
683681c189cf92972059d4e9
Junjie Wang
683681c189cf92972059d4ea
Xiaojun Jia
admin_assigned
2025-05-28T12:30:39.901Z
64c6627d5671d42e0adfad56
/avatars/8b98054b2911b86dcc4856a15306e60f.svg
jiaxiaojunQAQ
user
jiaxiaojunQAQ
683681c189cf92972059d4eb
Rupeng Zhang
claimed_verified
2025-07-16T15:07:36.096Z
62243b53e24db7b943abe2d2
https://cdn-avatars.huggingface.co/v1/production/uploads/1646541646787-noauth.png
Trevor Zhang
user
zrpxx
683681c189cf92972059d4ec
Mingyang Li
683681c189cf92972059d4ed
Zhe Liu
683681c189cf92972059d4ee
Yang Liu
683681c189cf92972059d4ef
Qing Wang
AdInject: Real-World Black-Box Attacks on Web Agents via Advertising
  Delivery
Vision-Language Model (VLM) based Web Agents represent a significant step
towards automating complex tasks by simulating human-like interaction with
websites. However, their deployment in uncontrolled web environments introduces
significant security vulnerabilities. Existing research on adversarial
environmental injection attacks often relies on unrealistic assumptions, such
as direct HTML manipulation, knowledge of user intent, or access to agent model
parameters, limiting their practical applicability. In this paper, we propose
AdInject, a novel and real-world black-box attack method that leverages the
internet advertising delivery to inject malicious content into the Web Agent's
environment. AdInject operates under a significantly more realistic threat
model than prior work, assuming a black-box agent, static malicious content
constraints, and no specific knowledge of user intent. AdInject includes
strategies for designing malicious ad content aimed at misleading agents into
clicking, and a VLM-based ad content optimization technique that infers
potential user intents from the target website's context and integrates these
intents into the ad content to make it appear more relevant or critical to the
agent's task, thus enhancing attack effectiveness. Experimental evaluations
demonstrate the effectiveness of AdInject, attack success rates exceeding 60%
in most scenarios and approaching 100% in certain cases. This strongly
demonstrates that prevalent advertising delivery constitutes a potent and
real-world vector for environment injection attacks against Web Agents. This
work highlights a critical vulnerability in Web Agent security arising from
real-world environment manipulation channels, underscoring the urgent need for
developing robust defense mechanisms against such threats. Our code is
available at https://github.com/NicerWang/AdInject.
AdInject is a novel real-world black-box attack method leveraging internet advertising to inject malicious content into vision-language model-based web agents, demonstrating significant vulnerability in web agent security.
vision-language model web agents adversarial environmental injection attacks AdInject internet advertising black-box agent static malicious content user intent attack success rates