Vision-language models (VLMs) have shown significant advancements in tasks
such as visual grounding, where they localize specific objects in images based
on natural language queries and images. However, security issues in visual
grounding tasks for VLMs remain underexplored, especially in the context of
backdoor attacks. In this paper, we introduce a novel input-aware backdoor
attack method, IAG, designed to manipulate the grounding behavior of VLMs. This
attack forces the model to ground a specific target object in the input image,
regardless of the user’s query. We propose an adaptive trigger generator that
embeds the semantic information of the attack target’s description into the
original image using a text-conditional U-Net, thereby overcoming the
open-vocabulary attack challenge. To ensure the attack’s stealthiness, we
utilize a reconstruction loss to minimize visual discrepancies between poisoned
and clean images. Additionally, we introduce a unified method for generating
attack data. IAG is evaluated theoretically and empirically, demonstrating its
feasibility and effectiveness. Notably, our ASR@0.5 on InternVL-2.5-8B reaches
over 65\% on various testing sets. IAG also shows promising potential on
manipulating Ferret-7B and LlaVA-1.5-7B with very little accuracy decrease on
clean samples. Extensive specific experiments, such as ablation study and
potential defense, also indicate the robustness and transferability of our
attack.