ZipVL: Efficient Large Vision-Language Models with Dynamic Token
  Sparsification and KV Cache Compression
The efficiency of large vision-language models (LVLMs) is constrained by the
computational bottleneck of the attention mechanism during the prefill phase
and the memory bottleneck of fetching the key-value (KV) cache in the decoding
phase, particularly in scenarios involving high-resolution images or videos.
Visual content often exhibits substantial redundancy, resulting in highly
sparse attention maps within LVLMs. This sparsity can be leveraged to
accelerate attention computation or compress the KV cache through various
approaches. However, most studies focus on addressing only one of these
bottlenecks and do not adequately support dynamic adjustment of sparsity
concerning distinct layers or tasks. In this paper, we present ZipVL, an
efficient inference framework designed for LVLMs that resolves both computation
and memory bottlenecks through a dynamic ratio allocation strategy of important
tokens. This ratio is adaptively determined based on the layer-specific
distribution of attention scores, rather than fixed hyper-parameters, thereby
improving efficiency for less complex tasks while maintaining high performance
for more challenging ones. Then we select important tokens based on their
normalized attention scores and perform attention mechanism solely on those
important tokens to accelerate the prefill phase. To mitigate the memory
bottleneck in the decoding phase, we employ mixed-precision quantization to the
KV cache, where high-bit quantization is used for caches of important tokens,
while low-bit quantization is applied to those of less importance. Our
experiments demonstrate that ZipVL can accelerate the prefill phase by
2.6times and reduce GPU memory usage by 50.0%, with a minimal accuracy
reduction of only 0.2% on Video-MME benchmark over LongVA-7B model, effectively
enhancing the generation efficiency of LVLMs.
670ca3f0977ff8095c05cb73
Yefei He
claimed_verified
2024-12-10T11:16:52.423Z
65a88c3d26598b995531fff1
/avatars/b1a524857d8572d0405476661b434160.svg
Yefei He
user
yefly
670ca3f0977ff8095c05cb74
Feng Chen
claimed_verified
2024-12-10T11:16:54.546Z
64574380f3ef144c0e69d484
/avatars/a0a84757cb0bf09c24291803e1389b49.svg
Feng Chen
user
chenfeng1271
670ca3f0977ff8095c05cb75
Jing Liu
670ca3f0977ff8095c05cb76
Wenqi Shao
670ca3f0977ff8095c05cb77
Hong Zhou
670ca3f0977ff8095c05cb78
Kaipeng Zhang
claimed_verified
2024-10-17T11:27:33.696Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
user
kpzhang996
670ca3f0977ff8095c05cb79
Bohan Zhuang
ZipVL: Efficient Large Vision-Language Models with Dynamic Token
  Sparsification and KV Cache Compression
The efficiency of large vision-language models (LVLMs) is constrained by the
computational bottleneck of the attention mechanism during the prefill phase
and the memory bottleneck of fetching the key-value (KV) cache in the decoding
phase, particularly in scenarios involving high-resolution images or videos.
Visual content often exhibits substantial redundancy, resulting in highly
sparse attention maps within LVLMs. This sparsity can be leveraged to
accelerate attention computation or compress the KV cache through various
approaches. However, most studies focus on addressing only one of these
bottlenecks and do not adequately support dynamic adjustment of sparsity
concerning distinct layers or tasks. In this paper, we present ZipVL, an
efficient inference framework designed for LVLMs that resolves both computation
and memory bottlenecks through a dynamic ratio allocation strategy of important
tokens. This ratio is adaptively determined based on the layer-specific
distribution of attention scores, rather than fixed hyper-parameters, thereby
improving efficiency for less complex tasks while maintaining high performance
for more challenging ones. Then we select important tokens based on their
normalized attention scores and perform attention mechanism solely on those
important tokens to accelerate the prefill phase. To mitigate the memory
bottleneck in the decoding phase, we employ mixed-precision quantization to the
KV cache, where high-bit quantization is used for caches of important tokens,
while low-bit quantization is applied to those of less importance. Our
experiments demonstrate that ZipVL can accelerate the prefill phase by
2.6times and reduce GPU memory usage by 50.0%, with a minimal accuracy
reduction of only 0.2% on Video-MME benchmark over LongVA-7B model, effectively
enhancing the generation efficiency of LVLMs.
ZipVL is an efficient inference framework for large vision-language models that addresses both computational and memory bottlenecks by dynamically allocating attention ratios and using mixed-precision quantization.
vision-language models attention mechanism key-value cache sparsity dynamic ratio allocation normalized attention scores mixed-precision quantization