Optimal Brain Restoration for Joint Quantization and Sparsification of
  LLMs
Recent advances in Large Language Model (LLM) compression, such as
quantization and pruning, have achieved notable success. However, as these
techniques gradually approach their respective limits, relying on a single
method for further compression has become increasingly challenging. In this
work, we explore an alternative solution by combining quantization and
sparsity. This joint approach, though promising, introduces new difficulties
due to the inherently conflicting requirements on weight distributions:
quantization favors compact ranges, while pruning benefits from high variance.
To attack this problem, we propose Optimal Brain Restoration (OBR), a general
and training-free framework that aligns pruning and quantization by error
compensation between both. OBR minimizes performance degradation on downstream
tasks by building on a second-order Hessian objective, which is then
reformulated into a tractable problem through surrogate approximation and
ultimately reaches a closed-form solution via group error compensation.
Experiments show that OBR enables aggressive W4A4KV4 quantization with 50%
sparsity on existing LLMs, and delivers up to 4.72x speedup and 6.4x memory
reduction compared to the FP16-dense baseline.
68ca5a906e0073c09bd1df7d
Hang Guo
claimed_verified
2025-09-17T12:47:25.533Z
67d6bb22eab66ce9cb4e3662
/avatars/814704eef9e6907d9d4ab407b605566b.svg
Hang Guo
user
HangGuo
68ca5a906e0073c09bd1df7e
Yawei Li
68ca5a906e0073c09bd1df7f
Luca Benini
Optimal Brain Restoration for Joint Quantization and Sparsification of
  LLMs
Recent advances in Large Language Model (LLM) compression, such as
quantization and pruning, have achieved notable success. However, as these
techniques gradually approach their respective limits, relying on a single
method for further compression has become increasingly challenging. In this
work, we explore an alternative solution by combining quantization and
sparsity. This joint approach, though promising, introduces new difficulties
due to the inherently conflicting requirements on weight distributions:
quantization favors compact ranges, while pruning benefits from high variance.
To attack this problem, we propose Optimal Brain Restoration (OBR), a general
and training-free framework that aligns pruning and quantization by error
compensation between both. OBR minimizes performance degradation on downstream
tasks by building on a second-order Hessian objective, which is then
reformulated into a tractable problem through surrogate approximation and
ultimately reaches a closed-form solution via group error compensation.
Experiments show that OBR enables aggressive W4A4KV4 quantization with 50%
sparsity on existing LLMs, and delivers up to 4.72x speedup and 6.4x memory
reduction compared to the FP16-dense baseline.
A framework combining quantization and pruning in LLMs through error compensation achieves significant speedup and memory reduction.
quantization pruning Optimal Brain Restoration (OBR) second-order Hessian objective surrogate approximation group error compensation W4A4KV4 quantization speedup memory reduction FP16-dense baseline