Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge
  Reasoning
Incorporating external knowledge in large language models (LLMs) enhances
their utility across diverse applications, but existing methods have
trade-offs. Retrieval-Augmented Generation (RAG) fetches evidence via
similarity search, but key information may fall outside top ranked results.
Long-context models can process multiple documents but are computationally
expensive and limited by context window size. Inspired by students condensing
study material for open-book exams, we propose task-aware key-value (KV) cache
compression, which compresses external knowledge in a zero- or few-shot setup.
This enables LLMs to reason efficiently over a compacted representation of all
relevant information. Experiments show our approach outperforms both RAG and
task-agnostic compression methods. On LongBench v2, it improves accuracy by up
to 7 absolute points over RAG with a 30x compression rate, while reducing
inference latency from 0.43s to 0.16s. A synthetic dataset highlights that RAG
performs well when sparse evidence suffices, whereas task-aware compression is
superior for broad knowledge tasks.
67d01f99a47b429698a2b34e
Giulio Corallo
claimed_verified
2025-03-11T11:43:02.035Z
62c88e75a5ac2974c0a5c8ea
https://cdn-avatars.huggingface.co/v1/production/uploads/1666106981514-62c88e75a5ac2974c0a5c8ea.jpeg
Giulio Corallo
user
giulio98
67d01f99a47b429698a2b34f
Orion Weller
admin_assigned
2025-03-11T14:17:49.755Z
6362d9712691058b19de1ba4
https://cdn-avatars.huggingface.co/v1/production/uploads/6362d9712691058b19de1ba4/c9QrA2oE6lcs_46ShaTY1.jpeg
Orion Weller
user
orionweller
67d01f99a47b429698a2b350
Fabio Petroni
admin_assigned
2025-03-11T14:17:56.241Z
6707ed4d371a647d6c8b5f04
/avatars/e9cb4c9caa3943588891e41e7f62cd69.svg
Fabio Petroni
user
fabiopetroni
67d01f99a47b429698a2b351
Paolo Papotti
extracted_confirmed
2025-03-11T12:35:45.283Z
67cfe779293bf2fc0c7163f2
/avatars/6d719eafd43a214d34d61bc6d9908511.svg
Paolo Papotti
user
papotti
Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge
  Reasoning
Incorporating external knowledge in large language models (LLMs) enhances
their utility across diverse applications, but existing methods have
trade-offs. Retrieval-Augmented Generation (RAG) fetches evidence via
similarity search, but key information may fall outside top ranked results.
Long-context models can process multiple documents but are computationally
expensive and limited by context window size. Inspired by students condensing
study material for open-book exams, we propose task-aware key-value (KV) cache
compression, which compresses external knowledge in a zero- or few-shot setup.
This enables LLMs to reason efficiently over a compacted representation of all
relevant information. Experiments show our approach outperforms both RAG and
task-agnostic compression methods. On LongBench v2, it improves accuracy by up
to 7 absolute points over RAG with a 30x compression rate, while reducing
inference latency from 0.43s to 0.16s. A synthetic dataset highlights that RAG
performs well when sparse evidence suffices, whereas task-aware compression is
superior for broad knowledge tasks.
Task-aware key-value cache compression for large language models enhances reasoning efficiency and accuracy over retrieval-augmented generation and task-agnostic compression methods.
Retrieval-Augmented Generation (RAG) long-context models key-value (KV) cache compression open-book exams LongBench v2 inference latency synthetic dataset broad knowledge tasks