Metis: Training Large Language Models with Advanced Low-Bit Quantization
This work identifies anisotropic parameter distributions as a fundamental
barrier to training large language models (LLMs) with low-bit quantization: a
few dominant singular values create wide numerical ranges that conflict with
the inherent bias of block-wise quantization. This bias disproportionately
preserves high-magnitude values while discarding smaller ones, causing training
instability and low model performance. This work introduces Metis, a training
framework that combines (i) spectral decomposition with random embedding to
efficiently disentangle dominant from long-tail components, compressing broad
distributions into quantization-friendly narrow ranges; (ii) adaptive learning
rates in the spectral domain to amplify underrepresented directions and better
capture diverse features critical for performance; and (iii) a dual-range
regularizer that jointly constrains numerical precision and parameter range
distribution, ensuring stable, unbiased low-bit training. With Metis, FP8
training surpasses FP32 baselines, and FP4 training achieves accuracy
comparable to FP32, paving the way for robust and scalable LLM training under
advanced low-bit quantization. The code implementation for Metis is available
at: https://github.com/typename-yyf/Metis-quantization.
68b7e917295f15ff609114ea
Hengjie Cao
68b7e917295f15ff609114eb
Mengyi Chen
admin_assigned
2025-09-03T08:50:58.093Z
65334842d34e9f02b98d9749
/avatars/5e6b06bb7fba729ae8f8368ed5458bc0.svg
Mengyi Chen
user
ChenMengyi
68b7e917295f15ff609114ec
Yifeng Yang
admin_assigned
2025-09-03T08:51:03.521Z
66541f311b1ce9f4a6c8e7a6
/avatars/1062a9887325c426863afa05620a81f8.svg
YIFENG YANG
user
cnyangyifeng
68b7e917295f15ff609114ed
Ruijun Huang
68b7e917295f15ff609114ee
Fang Dong
68b7e917295f15ff609114ef
Jixian Zhou
admin_assigned
2025-09-03T08:51:16.975Z
66e197e39656cebc87f17e3d
/avatars/84224455f762f7e9e7a7934532a29ac9.svg
Jixian Zhou
user
ZhouJixian
68b7e917295f15ff609114f0
Anrui Chen
68b7e917295f15ff609114f1
Mingzhi Dong
68b7e917295f15ff609114f2
Yujiang Wang
68b7e917295f15ff609114f3
Jinlong Hou
admin_assigned
2025-09-03T08:51:37.337Z
68aed07e662f89a73a744613
/avatars/2827a3703ea44c58c461c87e6bd48953.svg
jinlong hou
user
wenzi001
68b7e917295f15ff609114f4
Yuan Cheng
68b7e917295f15ff609114f5
Fan Wu
68b7e917295f15ff609114f6
Fan Yang
68b7e917295f15ff609114f7
Tun Lu
68b7e917295f15ff609114f8
Ning Gu
68b7e917295f15ff609114f9
Li Shang
Metis: Training Large Language Models with Advanced Low-Bit Quantization
This work identifies anisotropic parameter distributions as a fundamental
barrier to training large language models (LLMs) with low-bit quantization: a
few dominant singular values create wide numerical ranges that conflict with
the inherent bias of block-wise quantization. This bias disproportionately
preserves high-magnitude values while discarding smaller ones, causing training
instability and low model performance. This work introduces Metis, a training
framework that combines (i) spectral decomposition with random embedding to
efficiently disentangle dominant from long-tail components, compressing broad
distributions into quantization-friendly narrow ranges; (ii) adaptive learning
rates in the spectral domain to amplify underrepresented directions and better
capture diverse features critical for performance; and (iii) a dual-range
regularizer that jointly constrains numerical precision and parameter range
distribution, ensuring stable, unbiased low-bit training. With Metis, FP8
training surpasses FP32 baselines, and FP4 training achieves accuracy
comparable to FP32, paving the way for robust and scalable LLM training under
advanced low-bit quantization. The code implementation for Metis is available
at: https://github.com/typename-yyf/Metis-quantization.
Metis addresses training instability in low-bit quantized large language models by using spectral decomposition, adaptive learning rates, and dual-range regularization to improve performance and stability.
anisotropic parameter distributions large language models low-bit quantization singular values block-wise quantization spectral decomposition random embedding adaptive learning rates dual-range regularizer FP8 FP32