The Pitfalls of KV Cache Compression
KV cache compression promises increased throughput and efficiency with
negligible loss in performance. While the gains in throughput are indisputable
and recent literature has indeed shown minimal degradation on particular
benchmarks, in general the consequences of compression in realistic scenarios
such as multi-instruction prompting have been insufficiently studied. In this
paper, we identify several pitfalls practitioners should be aware of when
deploying KV cache compressed LLMs. Importantly, we show that certain
instructions degrade much more rapidly with compression, effectively causing
them to be completely ignored by the LLM. As a practical example of that, we
highlight system prompt leakage as a case study, empirically showing the impact
of compression on leakage and general instruction following. We show several
factors that play a role in prompt leakage: compression method, instruction
order, and KV eviction bias. We then propose simple changes to KV cache
eviction policies that can reduce the impact of these factors and improve the
overall performance in multi-instruction tasks.
68ddceaf6024653e8a3ed044
Alex Chen
claimed_verified
2025-10-02T03:08:03.866Z
65ac8b611f4b7b68fdc433e3
/avatars/37a4f122c2b8d21f481f1b5e3a2f4507.svg
Alex chen
user
ItisAlex
68ddceaf6024653e8a3ed045
Renato Geh
68ddceaf6024653e8a3ed046
Aditya Grover
68ddceaf6024653e8a3ed047
Guy Van den Broeck
68ddceaf6024653e8a3ed048
Daniel Israel
claimed_verified
2025-10-02T03:08:08.248Z
630139f1f6bea7dd15bdaf4e
/avatars/263536f6160f8c522d2a76ca2c4e4cc0.svg
Daniel Israel
user
danielmisrael
The Pitfalls of KV Cache Compression
KV cache compression promises increased throughput and efficiency with
negligible loss in performance. While the gains in throughput are indisputable
and recent literature has indeed shown minimal degradation on particular
benchmarks, in general the consequences of compression in realistic scenarios
such as multi-instruction prompting have been insufficiently studied. In this
paper, we identify several pitfalls practitioners should be aware of when
deploying KV cache compressed LLMs. Importantly, we show that certain
instructions degrade much more rapidly with compression, effectively causing
them to be completely ignored by the LLM. As a practical example of that, we
highlight system prompt leakage as a case study, empirically showing the impact
of compression on leakage and general instruction following. We show several
factors that play a role in prompt leakage: compression method, instruction
order, and KV eviction bias. We then propose simple changes to KV cache
eviction policies that can reduce the impact of these factors and improve the
overall performance in multi-instruction tasks.
Compression of KV caches in LLMs can lead to performance degradation in multi-instruction tasks, particularly affecting system prompt leakage, but improved eviction policies can mitigate these issues.
KV cache compression LLMs multi-instruction prompting system prompt leakage KV eviction policies