EpiCache: Episodic KV Cache Management for Long Conversational Question
  Answering
Recent advances in large language models (LLMs) have extended context
lengths, enabling assistants to sustain long histories for coherent,
personalized responses. This ability, however, hinges on Key-Value (KV)
caching, whose memory grows linearly with dialogue length and quickly dominates
under strict resource constraints. An active line of research for reducing this
overhead is KV cache compression, which seeks to limit cache size while
preserving accuracy. Yet existing methods face two major limitations: (i)
evicting entries after full-context prefill causes unbounded peak memory, and
(ii) query-dependent eviction narrows the cache to a single query, leading to
degraded accuracy in multi-turn conversations. We introduce EpiCache, a
training-free KV cache management framework for long conversational question
answering (LongConvQA) under fixed memory budgets. EpiCache bounds cache growth
through block-wise prefill and preserves topic-relevant context via episodic KV
compression, which clusters conversation history into coherent episodes and
applies episode-specific KV cache eviction. We further design an adaptive
layer-wise budget allocation strategy that measures each layer's sensitivity to
eviction and distributes the memory budget across layers accordingly. Across
three LongConvQA benchmarks, EpiCache improves accuracy by up to 40% over
recent baselines, sustains near-full KV accuracy under 4-6x compression, and
reduces latency and memory by up to 2.4x and 3.5x, thereby enabling efficient
multi-turn interaction under strict resource constraints.
68d1fc2d1ca7156988a8ec7e
Minsoo Kim
claimed_verified
2025-09-23T10:07:14.924Z
63c0e2503bdc86f8108da51b
/avatars/7d47f11992f030b3d831e45102581d1f.svg
Minsoo Kim
user
minsoo2333
68d1fc2d1ca7156988a8ec7f
Arnav Kundu
claimed_verified
2025-09-24T13:56:01.835Z
66289453c6da5c71a8b16b0d
/avatars/065a02d14ad8f085cf88ede87186e727.svg
Arnav Kundu
user
arnav1993k
68d1fc2d1ca7156988a8ec80
Han-Byul Kim
68d1fc2d1ca7156988a8ec81
Richa Dixit
68d1fc2d1ca7156988a8ec82
Minsik Cho
EpiCache: Episodic KV Cache Management for Long Conversational Question
  Answering
Recent advances in large language models (LLMs) have extended context
lengths, enabling assistants to sustain long histories for coherent,
personalized responses. This ability, however, hinges on Key-Value (KV)
caching, whose memory grows linearly with dialogue length and quickly dominates
under strict resource constraints. An active line of research for reducing this
overhead is KV cache compression, which seeks to limit cache size while
preserving accuracy. Yet existing methods face two major limitations: (i)
evicting entries after full-context prefill causes unbounded peak memory, and
(ii) query-dependent eviction narrows the cache to a single query, leading to
degraded accuracy in multi-turn conversations. We introduce EpiCache, a
training-free KV cache management framework for long conversational question
answering (LongConvQA) under fixed memory budgets. EpiCache bounds cache growth
through block-wise prefill and preserves topic-relevant context via episodic KV
compression, which clusters conversation history into coherent episodes and
applies episode-specific KV cache eviction. We further design an adaptive
layer-wise budget allocation strategy that measures each layer's sensitivity to
eviction and distributes the memory budget across layers accordingly. Across
three LongConvQA benchmarks, EpiCache improves accuracy by up to 40% over
recent baselines, sustains near-full KV accuracy under 4-6x compression, and
reduces latency and memory by up to 2.4x and 3.5x, thereby enabling efficient
multi-turn interaction under strict resource constraints.
EpiCache is a KV cache management framework for long conversational question answering that reduces memory usage and improves accuracy through block-wise prefill, episodic KV compression, and adaptive layer-wise budget allocation.
Key-Value (KV) caching KV cache compression block-wise prefill episodic KV compression adaptive layer-wise budget allocation LongConvQA multi-turn conversations