CompLLM: Compression for Long Context Q&A
Large Language Models (LLMs) face significant computational challenges when
processing long contexts due to the quadratic complexity of self-attention.
While soft context compression methods, which map input text to smaller latent
representations, have shown promise, their real-world adoption is limited.
Existing techniques typically compress the context as a single unit, which
leads to quadratic compression complexity and an inability to reuse
computations across queries with overlapping contexts. In this work, we
introduce CompLLM, a soft compression technique designed for practical
deployment. Instead of processing the context holistically, CompLLM divides it
into segments and compresses each one independently. This simple design choice
yields three critical properties: efficiency, as the compression step scales
linearly with the context length; scalability, enabling models trained on short
sequences (e.g., 1k tokens) to generalize to contexts of 100k tokens; and
reusability, allowing compressed segments to be cached and reused across
different queries. Our experiments show that with a 2x compression rate, at
high context lengths CompLLM speeds up Time To First Token (TTFT) by up to 4x
and reduces the KV cache size by 50%. Furthermore, CompLLM achieves performance
comparable to that obtained with the uncompressed context, and even surpasses
it on very long sequences, demonstrating its effectiveness and practical
utility.
68d4fdeaf9b028b804d3c227
Gabriele Berton
claimed_verified
2025-09-26T12:20:48.386Z
67a9e16f0710558f7bd8947a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/2SacRuI2bOsBgctaxWNGl.png
Gabriele Berton
user
gberton
68d4fdeaf9b028b804d3c228
Jayakrishnan Unnikrishnan
68d4fdeaf9b028b804d3c229
Son Tran
68d4fdeaf9b028b804d3c22a
Mubarak Shah
CompLLM: Compression for Long Context Q&A
Large Language Models (LLMs) face significant computational challenges when
processing long contexts due to the quadratic complexity of self-attention.
While soft context compression methods, which map input text to smaller latent
representations, have shown promise, their real-world adoption is limited.
Existing techniques typically compress the context as a single unit, which
leads to quadratic compression complexity and an inability to reuse
computations across queries with overlapping contexts. In this work, we
introduce CompLLM, a soft compression technique designed for practical
deployment. Instead of processing the context holistically, CompLLM divides it
into segments and compresses each one independently. This simple design choice
yields three critical properties: efficiency, as the compression step scales
linearly with the context length; scalability, enabling models trained on short
sequences (e.g., 1k tokens) to generalize to contexts of 100k tokens; and
reusability, allowing compressed segments to be cached and reused across
different queries. Our experiments show that with a 2x compression rate, at
high context lengths CompLLM speeds up Time To First Token (TTFT) by up to 4x
and reduces the KV cache size by 50%. Furthermore, CompLLM achieves performance
comparable to that obtained with the uncompressed context, and even surpasses
it on very long sequences, demonstrating its effectiveness and practical
utility.
CompLLM, a soft compression technique for LLMs, divides contexts into segments for efficient, scalable, and reusable compression, enhancing performance and reducing computational costs.
self-attention soft context compression latent representations CompLLM context segments Time To First Token KV cache