IAG: Input-aware Backdoor Attack on VLMs for Visual Grounding
Vision-language models (VLMs) have shown significant advancements in tasks
such as visual grounding, where they localize specific objects in images based
on natural language queries and images. However, security issues in visual
grounding tasks for VLMs remain underexplored, especially in the context of
backdoor attacks. In this paper, we introduce a novel input-aware backdoor
attack method, IAG, designed to manipulate the grounding behavior of VLMs. This
attack forces the model to ground a specific target object in the input image,
regardless of the user's query. We propose an adaptive trigger generator that
embeds the semantic information of the attack target's description into the
original image using a text-conditional U-Net, thereby overcoming the
open-vocabulary attack challenge. To ensure the attack's stealthiness, we
utilize a reconstruction loss to minimize visual discrepancies between poisoned
and clean images. Additionally, we introduce a unified method for generating
attack data. IAG is evaluated theoretically and empirically, demonstrating its
feasibility and effectiveness. Notably, our ASR@0.5 on InternVL-2.5-8B reaches
over 65\% on various testing sets. IAG also shows promising potential on
manipulating Ferret-7B and LlaVA-1.5-7B with very little accuracy decrease on
clean samples. Extensive specific experiments, such as ablation study and
potential defense, also indicate the robustness and transferability of our
attack.
689d555cb083e610d741e9fa
Junxian Li
claimed_verified
2025-08-14T13:37:30.188Z
656ae4088fb1ddf0d5ec9ac5
/avatars/e38468d2c0274f3c0f5732f30a2e3436.svg
Junxian Li
user
Duke-de-Artois
689d555cb083e610d741e9fb
Beining Xu
689d555cb083e610d741e9fc
Di Zhang
claimed_verified
2025-08-14T13:37:27.650Z
64bce15bafd1e46c5504ad38
https://cdn-avatars.huggingface.co/v1/production/uploads/64bce15bafd1e46c5504ad38/vkEjiu-mIagKlrXzDH75o.png
Di Zhang
user
di-zhang-fdu
IAG: Input-aware Backdoor Attack on VLMs for Visual Grounding
Vision-language models (VLMs) have shown significant advancements in tasks
such as visual grounding, where they localize specific objects in images based
on natural language queries and images. However, security issues in visual
grounding tasks for VLMs remain underexplored, especially in the context of
backdoor attacks. In this paper, we introduce a novel input-aware backdoor
attack method, IAG, designed to manipulate the grounding behavior of VLMs. This
attack forces the model to ground a specific target object in the input image,
regardless of the user's query. We propose an adaptive trigger generator that
embeds the semantic information of the attack target's description into the
original image using a text-conditional U-Net, thereby overcoming the
open-vocabulary attack challenge. To ensure the attack's stealthiness, we
utilize a reconstruction loss to minimize visual discrepancies between poisoned
and clean images. Additionally, we introduce a unified method for generating
attack data. IAG is evaluated theoretically and empirically, demonstrating its
feasibility and effectiveness. Notably, our ASR@0.5 on InternVL-2.5-8B reaches
over 65\% on various testing sets. IAG also shows promising potential on
manipulating Ferret-7B and LlaVA-1.5-7B with very little accuracy decrease on
clean samples. Extensive specific experiments, such as ablation study and
potential defense, also indicate the robustness and transferability of our
attack.
A novel input-aware backdoor attack method, IAG, manipulates vision-language models to ground specific objects in images regardless of user queries, using a text-conditional U-Net and reconstruction loss to ensure stealthiness.
vision-language models visual grounding backdoor attacks input-aware backdoor attack adaptive trigger generator text-conditional U-Net reconstruction loss ASR@0.5 InternVL-2.5-8B Ferret-7B LlaVA-1.5-7B ablation study potential defense