Inference-Time Hyper-Scaling with KV Cache Compression
Adrian Łańcucki, Konrad Staniszewski, Piotr Nawrot, Edoardo Maria Ponti
Inference-time scaling trades off efficiency for increased reasoning accuracy by generating longer or more parallel sequences. However, the cost of generation in Transformer LLMs is bottlenecked by the size of key-value (KV) cache, rather than the number of generated tokens. Hence, we explore inference-time hyper-scaling: by compressing the KV cache we can generate more tokens for the same compute budget, further improving the accuracy of scaled inference. The success of this approach, however, hinges on the ability of compression methods to maintain accuracy even under high compression ratios. To address this challenge, we introduce Dynamic Memory Sparsification (DMS), a novel and sample-efficient method for sparsifying KV caches. Instead of prematurely discarding cached tokens, DMS delays such decisions, thus implicitly merging representations and preserving critical information. We demonstrate the effectiveness of inference-time hyper-scaling on multiple families of LLMs, including DeepSeek R1-distilled models, showing that KV cache compression methods boost accuracy across various reasoning benchmarks for comparable inference runtime and memory load. Moreover, we find that DMS offers the best efficiency-accuracy trade-off among these methods.