Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language
  Models
Video large language models (VideoLLMs) have demonstrated the capability to
process longer video inputs and enable complex reasoning and analysis. However,
due to the thousands of visual tokens from the video frames, key-value (KV)
cache can significantly increase memory requirements, becoming a bottleneck for
inference speed and memory usage. KV cache quantization is a widely used
approach to address this problem. In this paper, we find that 2-bit KV
quantization of VideoLLMs can hardly hurt the model performance, while the
limit of KV cache quantization in even lower bits has not been investigated. To
bridge this gap, we introduce VidKV, a plug-and-play KV cache quantization
method to compress the KV cache to lower than 2 bits. Specifically, (1) for
key, we propose a mixed-precision quantization strategy in the channel
dimension, where we perform 2-bit quantization for anomalous channels and 1-bit
quantization combined with FFT for normal channels; (2) for value, we implement
1.58-bit quantization while selectively filtering semantically salient visual
tokens for targeted preservation, for a better trade-off between precision and
model performance. Importantly, our findings suggest that the value cache of
VideoLLMs should be quantized in a per-channel fashion instead of the per-token
fashion proposed by prior KV cache quantization works for LLMs. Empirically,
extensive results with LLaVA-OV-7B and Qwen2.5-VL-7B on six benchmarks show
that VidKV effectively compresses the KV cache to 1.5-bit and 1.58-bit
precision with almost no performance drop compared to the FP16 counterparts.
67dd2cbabf4c007db3bc0b76
Keda Tao
claimed_verified
2025-03-21T11:37:38.025Z
65ce28c6340c3e914285aa58
/avatars/ffaa6d6ce92274bff960f8ea229a37f8.svg
Keda TAO
user
KD-TAO
67dd2cbabf4c007db3bc0b77
Haoxuan You
admin_assigned
2025-03-21T12:59:36.878Z
65284ba3e3b3844f5e487a0b
/avatars/fce12a233bddb611d441f25a2ab17725.svg
Haoxuan You
user
HaoxuanYou
67dd2cbabf4c007db3bc0b78
Yang Sui
claimed_verified
2025-04-20T15:04:13.545Z
63787b13500186f250ba377c
/avatars/96bb6051662109e9cd25e6df7d738f17.svg
yangsui
user
yangsui
67dd2cbabf4c007db3bc0b79
Can Qin
admin_assigned
2025-03-21T12:59:48.645Z
66a9bdfc35c89692442ba4b7
/avatars/ad87d1d2d81775eeb0920bf3ebe08cc2.svg
Can Qin
user
canqin001
67dd2cbabf4c007db3bc0b7a
Huan Wang
claimed_verified
2025-05-28T10:11:29.992Z
62b624f3b52bef716e248fd7
https://cdn-avatars.huggingface.co/v1/production/uploads/62b624f3b52bef716e248fd7/AllcccKH-eBWduA8KVnOQ.png
Huan Wang
user
Huan-WhoRegisteredMyName
Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language
  Models
Video large language models (VideoLLMs) have demonstrated the capability to
process longer video inputs and enable complex reasoning and analysis. However,
due to the thousands of visual tokens from the video frames, key-value (KV)
cache can significantly increase memory requirements, becoming a bottleneck for
inference speed and memory usage. KV cache quantization is a widely used
approach to address this problem. In this paper, we find that 2-bit KV
quantization of VideoLLMs can hardly hurt the model performance, while the
limit of KV cache quantization in even lower bits has not been investigated. To
bridge this gap, we introduce VidKV, a plug-and-play KV cache quantization
method to compress the KV cache to lower than 2 bits. Specifically, (1) for
key, we propose a mixed-precision quantization strategy in the channel
dimension, where we perform 2-bit quantization for anomalous channels and 1-bit
quantization combined with FFT for normal channels; (2) for value, we implement
1.58-bit quantization while selectively filtering semantically salient visual
tokens for targeted preservation, for a better trade-off between precision and
model performance. Importantly, our findings suggest that the value cache of
VideoLLMs should be quantized in a per-channel fashion instead of the per-token
fashion proposed by prior KV cache quantization works for LLMs. Empirically,
extensive results with LLaVA-OV-7B and Qwen2.5-VL-7B on six benchmarks show
that VidKV effectively compresses the KV cache to 1.5-bit and 1.58-bit
precision with almost no performance drop compared to the FP16 counterparts.
VidKV, a new KV cache quantization method for VideoLLMs, compresses the KV cache to below 2 bits with minimal impact on performance by employing mixed-precision and channel-wise quantization techniques.
KV cache video frames visual tokens KV cache quantization VidKV mixed-precision quantization FFT per-channel quantization semantically salient visual tokens LLaVA-OV-7B Qwen2.5-VL-7B FP16