Attention Prompting on Image for Large Vision-Language Models
Compared with Large Language Models (LLMs), Large Vision-Language Models
(LVLMs) can also accept images as input, thus showcasing more interesting
emergent capabilities and demonstrating impressive performance on various
vision-language tasks. Motivated by text prompting in LLMs, visual prompting
has been explored to enhance LVLMs' capabilities of perceiving visual
information. However, previous visual prompting techniques solely process
visual inputs without considering text queries, limiting the models' ability to
follow text instructions to complete tasks. To fill this gap, in this work, we
propose a new prompting technique named Attention Prompting on Image, which
just simply overlays a text-query-guided attention heatmap on the original
input image and effectively enhances LVLM on various tasks. Specifically, we
generate an attention heatmap for the input image dependent on the text query
with an auxiliary model like CLIP. Then the heatmap simply multiplies the pixel
values of the original image to obtain the actual input image for the LVLM.
Extensive experiments on various vison-language benchmarks verify the
effectiveness of our technique. For example, Attention Prompting on Image
improves LLaVA-1.5 by 3.8% and 2.9% on MM-Vet and LLaVA-Wild benchmarks,
respectively.
66f4b50381a6b5bf75f11eba
Runpeng Yu
claimed_verified
2024-10-07T07:30:28.618Z
635364b3c41f548fe39db945
/avatars/ad1916bbfabca0b6651c8eabacc5eba8.svg
Runpeng Yu
user
rp-yu
66f4b50381a6b5bf75f11ebb
Weihao Yu
claimed_verified
2025-05-21T10:05:36.168Z
5df833bdda6d0311fd3d5403
https://cdn-avatars.huggingface.co/v1/production/uploads/5df833bdda6d0311fd3d5403/62OtGJEQXdOuhV9yCd4HS.png
Weihao Yu
user
whyu
66f4b50381a6b5bf75f11ebc
Xinchao Wang
Attention Prompting on Image for Large Vision-Language Models
Compared with Large Language Models (LLMs), Large Vision-Language Models
(LVLMs) can also accept images as input, thus showcasing more interesting
emergent capabilities and demonstrating impressive performance on various
vision-language tasks. Motivated by text prompting in LLMs, visual prompting
has been explored to enhance LVLMs' capabilities of perceiving visual
information. However, previous visual prompting techniques solely process
visual inputs without considering text queries, limiting the models' ability to
follow text instructions to complete tasks. To fill this gap, in this work, we
propose a new prompting technique named Attention Prompting on Image, which
just simply overlays a text-query-guided attention heatmap on the original
input image and effectively enhances LVLM on various tasks. Specifically, we
generate an attention heatmap for the input image dependent on the text query
with an auxiliary model like CLIP. Then the heatmap simply multiplies the pixel
values of the original image to obtain the actual input image for the LVLM.
Extensive experiments on various vison-language benchmarks verify the
effectiveness of our technique. For example, Attention Prompting on Image
improves LLaVA-1.5 by 3.8% and 2.9% on MM-Vet and LLaVA-Wild benchmarks,
respectively.
Attention Prompting on Image technique enhances Large Vision-Language Models by overlaying a text-query-guided attention heatmap on images, improving performance on vision-language tasks.
Large Language Models Large Vision-Language Models visual prompting attention heatmap CLIP