An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference
  Acceleration for Large Vision-Language Models
In this study, we identify the inefficient attention phenomena in Large
Vision-Language Models (LVLMs), notably within prominent models like LLaVA-1.5,
QwenVL-Chat and Video-LLaVA. We find out that the attention computation over
visual tokens is of extreme inefficiency in the deep layers of popular LVLMs,
suggesting a need for a sparser approach compared to textual data handling. To
this end, we introduce FastV, a versatile plug-and-play method designed to
optimize computational efficiency by learning adaptive attention patterns in
early layers and pruning visual tokens in subsequent ones. Our evaluations
demonstrate FastV's ability to dramatically reduce computational costs (e.g., a
45 reduction in FLOPs for LLaVA-1.5-13B) without sacrificing performance in a
wide range of image and video understanding tasks. The computational efficiency
and performance trade-off of FastV are highly customizable and
pareto-efficient. It can compress the FLOPs of a 13B-parameter model to achieve
a lower budget than that of a 7B-parameter model, while still maintaining
superior performance. We believe FastV has practical values for deployment of
LVLMs in edge devices and commercial models. Code is released at
https://github.com/pkunlp-icler/FastV.
65efce192392e7d5fb0bd36e
Liang Chen
claimed_verified
2024-03-12T12:06:02.476Z
61b0a4ce1b3d95b3d1ed9251
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Wwjr26vdudX5KYVTb8Q0a.png
Liang Chen
user
leonardPKU
65efce192392e7d5fb0bd36f
Haozhe Zhao
claimed_verified
2024-07-22T08:43:42.965Z
63ff09a02098b9ad105a09f6
/avatars/4409ca5d320050cf4c3df05962c7ff58.svg
Hans Zhao
user
BleachNick
65efce192392e7d5fb0bd370
Tianyu Liu
65efce192392e7d5fb0bd371
Shuai Bai
admin_assigned
2024-03-12T12:15:13.985Z
63451cf0a05b51f7ded25505
/avatars/dec4bbee4a82b773fc58dfc2dce9dbeb.svg
shuai bai
user
bluelike
65efce192392e7d5fb0bd372
Junyang Lin
admin_assigned
2024-03-12T12:15:20.851Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
65efce192392e7d5fb0bd373
Chang Zhou
admin_assigned
2024-03-12T12:15:34.914Z
622892774323cef93a956a4a
/avatars/e57ef5c3b0c4289988ccd42f14e54336.svg
chang zhou
user
jiemizc
65efce192392e7d5fb0bd374
Baobao Chang
An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference
  Acceleration for Large Vision-Language Models
In this study, we identify the inefficient attention phenomena in Large
Vision-Language Models (LVLMs), notably within prominent models like LLaVA-1.5,
QwenVL-Chat and Video-LLaVA. We find out that the attention computation over
visual tokens is of extreme inefficiency in the deep layers of popular LVLMs,
suggesting a need for a sparser approach compared to textual data handling. To
this end, we introduce FastV, a versatile plug-and-play method designed to
optimize computational efficiency by learning adaptive attention patterns in
early layers and pruning visual tokens in subsequent ones. Our evaluations
demonstrate FastV's ability to dramatically reduce computational costs (e.g., a
45 reduction in FLOPs for LLaVA-1.5-13B) without sacrificing performance in a
wide range of image and video understanding tasks. The computational efficiency
and performance trade-off of FastV are highly customizable and
pareto-efficient. It can compress the FLOPs of a 13B-parameter model to achieve
a lower budget than that of a 7B-parameter model, while still maintaining
superior performance. We believe FastV has practical values for deployment of
LVLMs in edge devices and commercial models. Code is released at
https://github.com/pkunlp-icler/FastV.
FastV optimizes Large Vision-Language Models by learning adaptive attention in early layers and pruning visual tokens to reduce computational costs without sacrificing performance.
Large Vision-Language Models LLaVA-1.5 QwenVL-Chat Video-LLaVA attention computation visual tokens sparser approach FastV adaptive attention patterns pruning FLOPs edge devices commercial models