CommVQ: Commutative Vector Quantization for KV Cache Compression
Large Language Models (LLMs) are increasingly used in applications requiring
long context lengths, but the key-value (KV) cache often becomes a memory
bottleneck on GPUs as context grows. To address this, we propose Commutative
Vector Quantization (CommVQ) to significantly reduce memory usage for
long-context LLM inference. We first introduce additive quantization with a
lightweight encoder and codebook to compress the KV cache, which can be decoded
via simple matrix multiplication. To further reduce computational costs during
decoding, we design the codebook to be commutative with Rotary Position
Embedding (RoPE) and train it using an Expectation-Maximization (EM) algorithm.
This enables efficient integration of decoding into the self-attention
mechanism. Our approach achieves high accuracy with additive quantization and
low overhead via the RoPE-commutative codebook. Experiments on long-context
benchmarks and GSM8K show that our method reduces FP16 KV cache size by 87.5%
with 2-bit quantization, while outperforming state-of-the-art KV cache
quantization methods. Notably, it enables 1-bit KV cache quantization with
minimal accuracy loss, allowing a LLaMA-3.1 8B model to run with a 128K context
length on a single RTX 4090 GPU. The source code is available at:
https://github.com/UMass-Embodied-AGI/CommVQ.
685a1a090e4ad7e21975859c
Junyan Li
admin_assigned
2025-06-24T13:34:41.316Z
62d09eb86a61a88ea0d83918
/avatars/81b511d94cced304ffca058caff662d4.svg
Junyan Li
user
senfu
685a1a090e4ad7e21975859d
Yang Zhang
685a1a090e4ad7e21975859e
Muhammad Yusuf Hassan
685a1a090e4ad7e21975859f
Talha Chafekar
admin_assigned
2025-06-24T13:35:02.649Z
6146f91079d11bc3c1714775
/avatars/418247a90388a59f5d1ee8899144b329.svg
Talha Chafekar
user
talha1503
685a1a090e4ad7e2197585a0
Tianle Cai
685a1a090e4ad7e2197585a1
Zhile Ren
685a1a090e4ad7e2197585a2
Pengsheng Guo
685a1a090e4ad7e2197585a3
Foroozan Karimzadeh
685a1a090e4ad7e2197585a4
Colorado Reed
685a1a090e4ad7e2197585a5
Chong Wang
685a1a090e4ad7e2197585a6
Chuang Gan
CommVQ: Commutative Vector Quantization for KV Cache Compression
Large Language Models (LLMs) are increasingly used in applications requiring
long context lengths, but the key-value (KV) cache often becomes a memory
bottleneck on GPUs as context grows. To address this, we propose Commutative
Vector Quantization (CommVQ) to significantly reduce memory usage for
long-context LLM inference. We first introduce additive quantization with a
lightweight encoder and codebook to compress the KV cache, which can be decoded
via simple matrix multiplication. To further reduce computational costs during
decoding, we design the codebook to be commutative with Rotary Position
Embedding (RoPE) and train it using an Expectation-Maximization (EM) algorithm.
This enables efficient integration of decoding into the self-attention
mechanism. Our approach achieves high accuracy with additive quantization and
low overhead via the RoPE-commutative codebook. Experiments on long-context
benchmarks and GSM8K show that our method reduces FP16 KV cache size by 87.5%
with 2-bit quantization, while outperforming state-of-the-art KV cache
quantization methods. Notably, it enables 1-bit KV cache quantization with
minimal accuracy loss, allowing a LLaMA-3.1 8B model to run with a 128K context
length on a single RTX 4090 GPU. The source code is available at:
https://github.com/UMass-Embodied-AGI/CommVQ.
Commutative Vector Quantization (CommVQ) reduces memory usage in long-context LLM inference by compressing the KV cache with additive quantization and integration of Rotary Position Embedding (RoPE).
Commutative Vector Quantization CommVQ additive quantization codebook Rotary Position Embedding RoPE Expectation-Maximization self-attention FP16 KV cache quantization GSM8K LLaMA-3.1 8B model RTX 4090 GPU