SINQ: Sinkhorn-Normalized Quantization for Calibration-Free
  Low-Precision LLM Weights
Post-training quantization has emerged as the most widely used strategy for
deploying large language models at low precision. Still, current methods show
perplexity degradation at bit-widths less than or equal to 4, partly because
representing outliers causes precision issues in parameters that share the same
scales as these outliers. This problem is especially pronounced for
calibration-free, uniform quantization methods. We introduce SINQ to augment
existing post-training quantizers with an additional second-axis scale factor
and a fast Sinkhorn-Knopp-style algorithm that finds scales to normalize
per-row and per-column variances, thereby minimizing a novel per-matrix proxy
target for quantization: the matrix imbalance. Our method has no interactions
between layers and can be trivially applied to new architectures to quantize
any linear layers. We evaluate our method on the Qwen3 model family and
DeepSeek-V2.5. SINQ improves WikiText2 and C4 perplexity significantly against
uncalibrated uniform quantization baselines and can be further enhanced by
combining it with calibration and non-uniform quantization levels. Code to
reproduce the results of this work and to easily quantize models using SINQ is
available at https://github.com/huawei-csl/SINQ.
68de4111a9129e507bf4a499
Lorenz K. Müller
claimed_verified
2025-10-03T12:53:10.505Z
6304c149dae2eb7d08407030
/avatars/efdca8cb990b16054970286382a2bbbd.svg
Lorenz Müller
user
lokamu
68de4111a9129e507bf4a49a
Philippe Bich
claimed_verified
2025-10-02T13:54:24.817Z
68b1e03b8aefe9d999b719f2
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/XyPVTSmon49qmwELaMrnX.png
Philippe Bich
user
pbicho
68de4111a9129e507bf4a49b
Jiawei Zhuang
68de4111a9129e507bf4a49c
Ahmet Çelik
claimed_verified
2025-10-02T13:54:21.246Z
6852cd91d1b6e201cd9c2446
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/btssGVVp_cFX7r_0wWV0y.png
Celik
user
storianc
68de4111a9129e507bf4a49d
Luca Benfenati
claimed_verified
2025-10-02T13:54:18.970Z
6548f86f46ce83d797a00699
/avatars/6dc0af3ee1587749313997fc46718cad.svg
Luca Benfenati
user
lucabnf
68de4111a9129e507bf4a49e
Lukas Cavigelli
claimed_verified
2025-10-03T12:53:08.178Z
6442ef61860f7a25bef0ea51
/avatars/8cccac5ac73498fa790e193a908c2057.svg
Lukas Cavigelli
user
lukasc-ch
SINQ: Sinkhorn-Normalized Quantization for Calibration-Free
  Low-Precision LLM Weights
Post-training quantization has emerged as the most widely used strategy for
deploying large language models at low precision. Still, current methods show
perplexity degradation at bit-widths less than or equal to 4, partly because
representing outliers causes precision issues in parameters that share the same
scales as these outliers. This problem is especially pronounced for
calibration-free, uniform quantization methods. We introduce SINQ to augment
existing post-training quantizers with an additional second-axis scale factor
and a fast Sinkhorn-Knopp-style algorithm that finds scales to normalize
per-row and per-column variances, thereby minimizing a novel per-matrix proxy
target for quantization: the matrix imbalance. Our method has no interactions
between layers and can be trivially applied to new architectures to quantize
any linear layers. We evaluate our method on the Qwen3 model family and
DeepSeek-V2.5. SINQ improves WikiText2 and C4 perplexity significantly against
uncalibrated uniform quantization baselines and can be further enhanced by
combining it with calibration and non-uniform quantization levels. Code to
reproduce the results of this work and to easily quantize models using SINQ is
available at https://github.com/huawei-csl/SINQ.
SINQ enhances post-training quantization by introducing a second-axis scale factor and Sinkhorn-Knopp-style algorithm to minimize matrix imbalance, improving perplexity on large language models.
post-training quantization large language models perplexity calibration-free uniform quantization second-axis scale factor Sinkhorn-Knopp-style algorithm matrix imbalance Qwen3 DeepSeek-V2.5 WikiText2 C4 non-uniform quantization levels