ChunkKV: Semantic-Preserving KV Cache Compression for Efficient
  Long-Context LLM Inference
To reduce memory costs in long-context inference with Large Language Models
(LLMs), many recent works focus on compressing the key-value (KV) cache of
different tokens. However, we identify that the previous KV cache compression
methods measure token importance individually, neglecting the dependency
between different tokens in the real-world language characterics. In light of
this, we introduce ChunkKV, grouping the tokens in a chunk as a basic
compressing unit, and retaining the most informative semantic chunks while
discarding the less important ones. Furthermore, observing that ChunkKV
exhibits higher similarity in the preserved indices across different layers, we
propose layer-wise index reuse to further reduce computational overhead. We
evaluated ChunkKV on cutting-edge long-context benchmarks including LongBench
and Needle-In-A-HayStack, as well as the GSM8K and JailbreakV in-context
learning benchmark. Our experiments with instruction tuning and multi-step
reasoning (O1 and R1) LLMs, achieve up to 10\% performance improvement under
aggressive compression ratios compared to existing methods.
67a1779d5f583199ce7921ad
Xiang Liu
claimed_verified
2025-02-04T14:48:39.990Z
63024676056ec3a2a8714b24
https://cdn-avatars.huggingface.co/v1/production/uploads/1661093436322-noauth.jpeg
Xiang Liu
user
Dominic789654
67a1779d5f583199ce7921ae
Zhenheng Tang
67a1779d5f583199ce7921af
Peijie Dong
claimed_verified
2025-06-02T07:54:08.189Z
6395f845aec00abff778ad31
https://cdn-avatars.huggingface.co/v1/production/uploads/6395f845aec00abff778ad31/bZkAlchSvqER1HgBKmcHI.jpeg
PeijieDong
user
pprp
67a1779d5f583199ce7921b0
Zeyu Li
67a1779d5f583199ce7921b1
Bo Li
67a1779d5f583199ce7921b2
Xuming Hu
67a1779d5f583199ce7921b3
Xiaowen Chu
ChunkKV: Semantic-Preserving KV Cache Compression for Efficient
  Long-Context LLM Inference
To reduce memory costs in long-context inference with Large Language Models
(LLMs), many recent works focus on compressing the key-value (KV) cache of
different tokens. However, we identify that the previous KV cache compression
methods measure token importance individually, neglecting the dependency
between different tokens in the real-world language characterics. In light of
this, we introduce ChunkKV, grouping the tokens in a chunk as a basic
compressing unit, and retaining the most informative semantic chunks while
discarding the less important ones. Furthermore, observing that ChunkKV
exhibits higher similarity in the preserved indices across different layers, we
propose layer-wise index reuse to further reduce computational overhead. We
evaluated ChunkKV on cutting-edge long-context benchmarks including LongBench
and Needle-In-A-HayStack, as well as the GSM8K and JailbreakV in-context
learning benchmark. Our experiments with instruction tuning and multi-step
reasoning (O1 and R1) LLMs, achieve up to 10\% performance improvement under
aggressive compression ratios compared to existing methods.
ChunkKV improves long-context inference in Large Language Models by compressing semantic chunks of tokens and reusing indices across layers, enhancing performance under high compression.
ChunkKV key-value cache token importance semantic chunks layer-wise index reuse LongBench Needle-In-A-HayStack GSM8K JailbreakV instruction tuning multi-step reasoning