A Simple and Effective L_2 Norm-Based Strategy for KV Cache
  Compression
The deployment of large language models (LLMs) is often hindered by the
extensive memory requirements of the Key-Value (KV) cache, especially as
context lengths increase. Existing approaches to reduce the KV cache size
involve either fine-tuning the model to learn a compression strategy or
leveraging attention scores to reduce the sequence length. We analyse the
attention distributions in decoder-only Transformers-based models and observe
that attention allocation patterns stay consistent across most layers.
Surprisingly, we find a clear correlation between the L_2 and the attention
scores over cached KV pairs, where a low L_2 of a key embedding usually leads
to a high attention score during decoding. This finding indicates that the
influence of a KV pair is potentially determined by the key embedding itself
before being queried. Based on this observation, we compress the KV cache based
on the L_2 of key embeddings. Our experimental results show that this simple
strategy can reduce the KV cache size by 50% on language modelling and
needle-in-a-haystack tasks and 90% on passkey retrieval tasks without losing
accuracy.
66716ba1d059806ae6e4ba43
Alessio Devoto
claimed_verified
2024-10-30T09:56:26.707Z
62180288d05d07ebf7de5f31
https://cdn-avatars.huggingface.co/v1/production/uploads/62180288d05d07ebf7de5f31/io2RsaRSjIDktQylXVvEB.jpeg
Alessio
user
alessiodevoto
66716ba1d059806ae6e4ba44
Yu Zhao
claimed_verified
2024-06-18T15:55:05.167Z
654a500d9b8bd6406d431c0d
/avatars/a6b76441bbc6f4b71d49c52e454c9ef7.svg
Yu Zhao
user
yuzhaouoe
66716ba1d059806ae6e4ba45
Simone Scardapane
claimed_verified
2024-12-21T15:20:24.561Z
6321a810a97afe3c4c64457f
/avatars/769bd3874163823733c817d8707e279f.svg
Simone Scardapane
user
sscardapane
66716ba1d059806ae6e4ba46
Pasquale Minervini
claimed_verified
2024-06-18T16:29:44.308Z
61001311e043e15c13412d30
https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png
Pasquale Minervini
user
pminervini
A Simple and Effective L_2 Norm-Based Strategy for KV Cache
  Compression
The deployment of large language models (LLMs) is often hindered by the
extensive memory requirements of the Key-Value (KV) cache, especially as
context lengths increase. Existing approaches to reduce the KV cache size
involve either fine-tuning the model to learn a compression strategy or
leveraging attention scores to reduce the sequence length. We analyse the
attention distributions in decoder-only Transformers-based models and observe
that attention allocation patterns stay consistent across most layers.
Surprisingly, we find a clear correlation between the L_2 and the attention
scores over cached KV pairs, where a low L_2 of a key embedding usually leads
to a high attention score during decoding. This finding indicates that the
influence of a KV pair is potentially determined by the key embedding itself
before being queried. Based on this observation, we compress the KV cache based
on the L_2 of key embeddings. Our experimental results show that this simple
strategy can reduce the KV cache size by 50% on language modelling and
needle-in-a-haystack tasks and 90% on passkey retrieval tasks without losing
accuracy.
By compressing the KV cache based on the $L_2$ norm of key embeddings, the sizes required for transformer models can be significantly reduced without sacrificing accuracy.
large language models key-value cache decoder-only Transformers attention distributions $L_2$ norm key embedding language modelling needle-in-a-haystack tasks passkey retrieval tasks