VisionZip: Longer is Better but Not Necessary in Vision Language Models
Recent advancements in vision-language models have enhanced performance by
increasing the length of visual tokens, making them much longer than text
tokens and significantly raising computational costs. However, we observe that
the visual tokens generated by popular vision encoders, such as CLIP and
SigLIP, contain significant redundancy. To address this, we introduce
VisionZip, a simple yet effective method that selects a set of informative
tokens for input to the language model, reducing visual token redundancy and
improving efficiency while maintaining model performance. The proposed
VisionZip can be widely applied to image and video understanding tasks and is
well-suited for multi-turn dialogues in real-world scenarios, where previous
methods tend to underperform. Experimental results show that VisionZip
outperforms the previous state-of-the-art method by at least 5% performance
gains across nearly all settings. Moreover, our method significantly enhances
model inference speed, improving the prefilling time by 8x and enabling the
LLaVA-Next 13B model to infer faster than the LLaVA-Next 7B model while
achieving better results. Furthermore, we analyze the causes of this redundancy
and encourage the community to focus on extracting better visual features
rather than merely increasing token length. Our code is available at
https://github.com/dvlab-research/VisionZip .
6752642e9106cbed4a680b9d
Senqiao Yang
claimed_verified
2024-12-06T09:09:21.290Z
6527b7280ae663e384eb8499
https://cdn-avatars.huggingface.co/v1/production/uploads/6527b7280ae663e384eb8499/73yF3eu2cUx7jVZrhXnXx.jpeg
Senqiao Yang
user
Senqiao
6752642e9106cbed4a680b9e
Yukang Chen
admin_assigned
2024-12-06T09:45:09.922Z
65b31d77f32c79fea79ba4d1
/avatars/8c51c1ff337ac372c7caafa4fa5019dc.svg
Yukang Chen
user
Yukang2
6752642e9106cbed4a680b9f
Zhuotao Tian
admin_assigned
2024-12-06T09:45:21.690Z
64d104a37a7305c5895bd720
/avatars/2d9eff3a2dff6d02e45ae8964fb91f27.svg
zt tian
user
tianzhuotao
6752642e9106cbed4a680ba0
Chengyao Wang
admin_assigned
2024-12-06T09:45:28.672Z
64c3257a027e6d966673712a
/avatars/abac7539c8e90c2780ecc445536d8c2c.svg
Chengyao Wang
user
wangchengyao
6752642e9106cbed4a680ba1
Jingyao Li
admin_assigned
2024-12-06T09:45:35.074Z
647ff779d037515be1169e35
/avatars/741096302cbdeb3670a8f31cf2d31d5d.svg
Jingyao Li
user
JingyaoLi
6752642e9106cbed4a680ba2
Bei Yu
6752642e9106cbed4a680ba3
Jiaya Jia
VisionZip: Longer is Better but Not Necessary in Vision Language Models
Recent advancements in vision-language models have enhanced performance by
increasing the length of visual tokens, making them much longer than text
tokens and significantly raising computational costs. However, we observe that
the visual tokens generated by popular vision encoders, such as CLIP and
SigLIP, contain significant redundancy. To address this, we introduce
VisionZip, a simple yet effective method that selects a set of informative
tokens for input to the language model, reducing visual token redundancy and
improving efficiency while maintaining model performance. The proposed
VisionZip can be widely applied to image and video understanding tasks and is
well-suited for multi-turn dialogues in real-world scenarios, where previous
methods tend to underperform. Experimental results show that VisionZip
outperforms the previous state-of-the-art method by at least 5% performance
gains across nearly all settings. Moreover, our method significantly enhances
model inference speed, improving the prefilling time by 8x and enabling the
LLaVA-Next 13B model to infer faster than the LLaVA-Next 7B model while
achieving better results. Furthermore, we analyze the causes of this redundancy
and encourage the community to focus on extracting better visual features
rather than merely increasing token length. Our code is available at
https://github.com/dvlab-research/VisionZip .
VisionZip reduces visual token redundancy and enhances efficiency in vision-language models, improving performance and inference speed across various tasks.
vision-language models visual tokens text tokens CLIP SigLIP VisionZip image and video understanding tasks multi-turn dialogues prefilling time LLaVA-Next