SnapKV: LLM Knows What You are Looking for Before Generation
Large Language Models (LLMs) have made remarkable progress in processing
extensive contexts, with the Key-Value (KV) cache playing a vital role in
enhancing their performance. However, the growth of the KV cache in response to
increasing input length poses challenges to memory and time efficiency. To
address this problem, this paper introduces SnapKV, an innovative and
fine-tuning-free approach that efficiently minimizes KV cache size while still
delivering comparable performance in real-world applications.
  We discover that each attention head in the model consistently focuses on
specific prompt attention features during generation. Meanwhile, this robust
pattern can be obtained from an `observation' window located at the end of the
prompts. Drawing on this insight, SnapKV automatically compresses KV caches by
selecting clustered important KV positions for each attention head. Our
approach significantly reduces the growing computational overhead and memory
footprint when processing long input sequences. Specifically, SnapKV achieves a
consistent decoding speed with a 3.6x increase in generation speed and an 8.2x
enhancement in memory efficiency compared to baseline when processing inputs of
16K tokens. At the same time, it maintains comparable performance to baseline
models across 16 long sequence datasets. Moreover, SnapKV can process up to
380K context tokens on a single A100-80GB GPU using HuggingFace implementation
with minor changes, exhibiting only a negligible accuracy drop in the
Needle-in-a-Haystack test. Further comprehensive studies suggest SnapKV's
potential for practical applications.
662861ec650b96e3ab6ae49d
Yuhong Li
662861ec650b96e3ab6ae49e
Yingbing Huang
662861ec650b96e3ab6ae49f
Bowen Yang
662861ec650b96e3ab6ae4a0
Bharat Venkitesh
admin_assigned
2024-04-24T11:39:34.088Z
65e8e96868cd34406483a639
/avatars/77fbe9ebd1fccd2e15abb403e2139652.svg
Bharat venkitesh
user
bharatv007
662861ec650b96e3ab6ae4a1
Acyr Locatelli
admin_assigned
2024-04-24T11:39:39.937Z
60d370b613f774189902f556
/avatars/717bd7955f30eb76310d2074fd51afce.svg
Acyr Locatelli
user
acyrl
662861ec650b96e3ab6ae4a2
Hanchen Ye
662861ec650b96e3ab6ae4a3
Tianle Cai
admin_assigned
2024-04-24T11:40:03.968Z
64459357c525660aa20be337
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/1fryBXRuSvAgMIDkpWBpI.jpeg
Tianle Cai
user
tianlecai
662861ec650b96e3ab6ae4a4
Patrick Lewis
662861ec650b96e3ab6ae4a5
Deming Chen
SnapKV: LLM Knows What You are Looking for Before Generation
Large Language Models (LLMs) have made remarkable progress in processing
extensive contexts, with the Key-Value (KV) cache playing a vital role in
enhancing their performance. However, the growth of the KV cache in response to
increasing input length poses challenges to memory and time efficiency. To
address this problem, this paper introduces SnapKV, an innovative and
fine-tuning-free approach that efficiently minimizes KV cache size while still
delivering comparable performance in real-world applications.
  We discover that each attention head in the model consistently focuses on
specific prompt attention features during generation. Meanwhile, this robust
pattern can be obtained from an `observation' window located at the end of the
prompts. Drawing on this insight, SnapKV automatically compresses KV caches by
selecting clustered important KV positions for each attention head. Our
approach significantly reduces the growing computational overhead and memory
footprint when processing long input sequences. Specifically, SnapKV achieves a
consistent decoding speed with a 3.6x increase in generation speed and an 8.2x
enhancement in memory efficiency compared to baseline when processing inputs of
16K tokens. At the same time, it maintains comparable performance to baseline
models across 16 long sequence datasets. Moreover, SnapKV can process up to
380K context tokens on a single A100-80GB GPU using HuggingFace implementation
with minor changes, exhibiting only a negligible accuracy drop in the
Needle-in-a-Haystack test. Further comprehensive studies suggest SnapKV's
potential for practical applications.
SnapKV reduces KV cache size and computational overhead in LLMs by compressing the cache based on observed attention patterns, improving decoding speed and memory efficiency without significant performance loss.
Large Language Models Key-Value cache SnapKV attention heads prompt attention features observation window attention positions computational overhead memory footprint decoding speed generation speed memory efficiency Needle-in-a-Haystack test