XQuant: Breaking the Memory Wall for LLM Inference with KV Cache
  Rematerialization
Although LLM inference has emerged as a critical workload for many downstream
applications, efficiently inferring LLMs is challenging due to the substantial
memory footprint and bandwidth requirements. In parallel, compute capabilities
have steadily outpaced both memory capacity and bandwidth over the last few
decades, a trend that remains evident in modern GPU hardware and exacerbates
the challenge of LLM inference. As such, new algorithms are emerging that trade
increased computation for reduced memory operations. To that end, we present
XQuant, which takes advantage of this trend, enabling an order-of-magnitude
reduction in memory consumption through low-bit quantization with substantial
accuracy benefits relative to state-of-the-art KV cache quantization methods.
We accomplish this by quantizing and caching the layer input activations X,
instead of using standard KV caching, and then rematerializing the Keys and
Values on-the-fly during inference. This results in an immediate 2times
memory savings compared to KV caching. By applying XQuant, we achieve up to
sim 7.7times memory savings with <0.1 perplexity degradation compared to
the FP16 baseline. Furthermore, our approach leverages the fact that X values
are similar across layers. Building on this observation, we introduce
XQuant-CL, which exploits the cross-layer similarity in the X embeddings for
extreme compression. Across different models, XQuant-CL attains up to
10times memory savings relative to the FP16 baseline with only 0.01
perplexity degradation, and 12.5times memory savings with only 0.1
perplexity degradation. XQuant exploits the rapidly increasing compute
capabilities of hardware platforms to eliminate the memory bottleneck, while
surpassing state-of-the-art KV cache quantization methods and achieving
near-FP16 accuracy across a wide range of models.
68a1e494a4caabb4320e6268
Aditya Tomar
claimed_verified
2025-08-19T10:49:28.964Z
65fe49de871b36bf84c0ba05
/avatars/0fe082518fb9ea40e23414c83ee5043e.svg
Aditya Tomar
user
adityastomar
68a1e494a4caabb4320e6269
Coleman Hooper
68a1e494a4caabb4320e626a
Minjae Lee
claimed_verified
2025-08-26T09:50:48.832Z
6788cbfd6cdca10c9bb3dea5
/avatars/a0084b72d5b1a82d9c8a5aef1a230ffd.svg
Minjae Lee
user
FuriosaMJLEE
68a1e494a4caabb4320e626b
Haocheng Xi
68a1e494a4caabb4320e626c
Rishabh Tiwari
claimed_verified
2025-08-19T10:49:21.586Z
66db38c38d2688295f731283
/avatars/a1f832d354a1f5d5c11593bf276b47a6.svg
Rishabh Tiwari
user
rishabh2k1
68a1e494a4caabb4320e626d
Wonjun Kang
claimed_verified
2025-08-19T10:49:23.686Z
63e1d5247fbb6ae4d4f4cc8e
/avatars/8a8f700adf9e8000641c2c2f6bd56080.svg
Wonjun Kang
user
wjkang
68a1e494a4caabb4320e626e
Luca Manolache
68a1e494a4caabb4320e626f
Michael W. Mahoney
68a1e494a4caabb4320e6270
Kurt Keutzer
68a1e494a4caabb4320e6271
Amir Gholami
XQuant: Breaking the Memory Wall for LLM Inference with KV Cache
  Rematerialization
Although LLM inference has emerged as a critical workload for many downstream
applications, efficiently inferring LLMs is challenging due to the substantial
memory footprint and bandwidth requirements. In parallel, compute capabilities
have steadily outpaced both memory capacity and bandwidth over the last few
decades, a trend that remains evident in modern GPU hardware and exacerbates
the challenge of LLM inference. As such, new algorithms are emerging that trade
increased computation for reduced memory operations. To that end, we present
XQuant, which takes advantage of this trend, enabling an order-of-magnitude
reduction in memory consumption through low-bit quantization with substantial
accuracy benefits relative to state-of-the-art KV cache quantization methods.
We accomplish this by quantizing and caching the layer input activations X,
instead of using standard KV caching, and then rematerializing the Keys and
Values on-the-fly during inference. This results in an immediate 2times
memory savings compared to KV caching. By applying XQuant, we achieve up to
sim 7.7times memory savings with <0.1 perplexity degradation compared to
the FP16 baseline. Furthermore, our approach leverages the fact that X values
are similar across layers. Building on this observation, we introduce
XQuant-CL, which exploits the cross-layer similarity in the X embeddings for
extreme compression. Across different models, XQuant-CL attains up to
10times memory savings relative to the FP16 baseline with only 0.01
perplexity degradation, and 12.5times memory savings with only 0.1
perplexity degradation. XQuant exploits the rapidly increasing compute
capabilities of hardware platforms to eliminate the memory bottleneck, while
surpassing state-of-the-art KV cache quantization methods and achieving
near-FP16 accuracy across a wide range of models.
XQuant and XQuant-CL reduce memory consumption in LLM inference through low-bit quantization and cross-layer similarity exploitation, achieving significant memory savings with minimal accuracy loss.
LLM inference low-bit quantization KV cache quantization layer input activations rematerialization XQuant XQuant-CL cross-layer similarity memory savings perplexity degradation FP16 baseline