Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression
Autoregressive language models rely on a Key-Value (KV) Cache, which avoids
re-computing past hidden states during generation, making it faster. As model
sizes and context lengths grow, the KV Cache becomes a significant memory
bottleneck, which calls for compression methods that limit its size during
generation. In this paper, we discover surprising properties of Query (Q) and
Key (K) vectors that allow us to efficiently approximate attention scores
without computing the attention maps. We propose Q-Filters, a training-free KV
Cache compression method that filters out less crucial Key-Value pairs based on
a single context-agnostic projection. Contrarily to many alternatives,
Q-Filters is compatible with FlashAttention, as it does not require direct
access to attention weights. Experimental results in long-context settings
demonstrate that Q-Filters is competitive with attention-based compression
methods such as SnapKV in retrieval tasks while consistently outperforming
efficient compression schemes such as Streaming-LLM in generation setups.
Notably, Q-Filters achieves a 99% accuracy in the needle-in-a-haystack task
with a x32 compression level while reducing the generation perplexity drop by
up to 65% in text generation compared to Streaming-LLM.
67c87b12d62973a5d9f014b3
Nathan Godey
extracted_pending
2025-03-05T16:25:56.974Z
61d2d6a5291690e1c7b4dd2d
https://cdn-avatars.huggingface.co/v1/production/uploads/1666688353023-61d2d6a5291690e1c7b4dd2d.jpeg
Nathan Godey
user
nthngdy
67c87b12d62973a5d9f014b4
Alessio Devoto
67c87b12d62973a5d9f014b5
Yu Zhao
claimed_verified
2025-03-06T13:34:20.584Z
654a500d9b8bd6406d431c0d
/avatars/a6b76441bbc6f4b71d49c52e454c9ef7.svg
Yu Zhao
user
yuzhaouoe
67c87b12d62973a5d9f014b6
Simone Scardapane
67c87b12d62973a5d9f014b7
Pasquale Minervini
67c87b12d62973a5d9f014b8
Éric de la Clergerie
67c87b12d62973a5d9f014b9
Benoît Sagot
Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression
Autoregressive language models rely on a Key-Value (KV) Cache, which avoids
re-computing past hidden states during generation, making it faster. As model
sizes and context lengths grow, the KV Cache becomes a significant memory
bottleneck, which calls for compression methods that limit its size during
generation. In this paper, we discover surprising properties of Query (Q) and
Key (K) vectors that allow us to efficiently approximate attention scores
without computing the attention maps. We propose Q-Filters, a training-free KV
Cache compression method that filters out less crucial Key-Value pairs based on
a single context-agnostic projection. Contrarily to many alternatives,
Q-Filters is compatible with FlashAttention, as it does not require direct
access to attention weights. Experimental results in long-context settings
demonstrate that Q-Filters is competitive with attention-based compression
methods such as SnapKV in retrieval tasks while consistently outperforming
efficient compression schemes such as Streaming-LLM in generation setups.
Notably, Q-Filters achieves a 99% accuracy in the needle-in-a-haystack task
with a x32 compression level while reducing the generation perplexity drop by
up to 65% in text generation compared to Streaming-LLM.
Q-Filters is a KV Cache compression method using Q-Filters that efficiently reduces memory usage without compromising attention scores and text generation quality.
Key-Value Cache autoregressive language models Query vectors Key vectors attention scores FlashAttention SnapKV Streaming-LLM needle-in-a-haystack task generation perplexity