Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large
  Language Models
Extreme activation outliers in Large Language Models (LLMs) critically
degrade quantization performance, hindering efficient on-device deployment.
While channel-wise operations and adaptive gradient scaling are recognized
causes, practical mitigation remains challenging. We introduce Outlier-Safe
Pre-Training (OSP), a practical guideline that proactively prevents outlier
formation rather than relying on post-hoc mitigation. OSP combines three key
innovations: (1) the Muon optimizer, eliminating privileged bases while
maintaining training efficiency; (2) Single-Scale RMSNorm, preventing
channel-wise amplification; and (3) a learnable embedding projection,
redistributing activation magnitudes originating from embedding matrices. We
validate OSP by training a 1.4B-parameter model on 1 trillion tokens, which is
the first production-scale LLM trained without such outliers. Under aggressive
4-bit quantization, our OSP model achieves a 35.7 average score across 10
benchmarks (compared to 26.5 for an Adam-trained model), with only a 2%
training overhead. Remarkably, OSP models exhibit near-zero excess kurtosis
(0.04) compared to extreme values (1818.56) in standard models, fundamentally
altering LLM quantization behavior. Our work demonstrates that outliers are not
inherent to LLMs but are consequences of training strategies, paving the way
for more efficient LLM deployment. The source code and pretrained checkpoints
are available at https://github.com/dmis-lab/Outlier-Safe-Pre-Training.
685c1546df8a0d6c70bbf94e
Jungwoo Park
claimed_verified
2025-06-26T09:32:38.469Z
60f8435644e75317cc02ed51
/avatars/68b7fc077fe2bda6607b1c470add8140.svg
Jungwoo Park
user
affjljoo3581
685c1546df8a0d6c70bbf94f
Taewhoo Lee
685c1546df8a0d6c70bbf950
Chanwoong Yoon
685c1546df8a0d6c70bbf951
Hyeon Hwang
685c1546df8a0d6c70bbf952
Jaewoo Kang
Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large
  Language Models
Extreme activation outliers in Large Language Models (LLMs) critically
degrade quantization performance, hindering efficient on-device deployment.
While channel-wise operations and adaptive gradient scaling are recognized
causes, practical mitigation remains challenging. We introduce Outlier-Safe
Pre-Training (OSP), a practical guideline that proactively prevents outlier
formation rather than relying on post-hoc mitigation. OSP combines three key
innovations: (1) the Muon optimizer, eliminating privileged bases while
maintaining training efficiency; (2) Single-Scale RMSNorm, preventing
channel-wise amplification; and (3) a learnable embedding projection,
redistributing activation magnitudes originating from embedding matrices. We
validate OSP by training a 1.4B-parameter model on 1 trillion tokens, which is
the first production-scale LLM trained without such outliers. Under aggressive
4-bit quantization, our OSP model achieves a 35.7 average score across 10
benchmarks (compared to 26.5 for an Adam-trained model), with only a 2%
training overhead. Remarkably, OSP models exhibit near-zero excess kurtosis
(0.04) compared to extreme values (1818.56) in standard models, fundamentally
altering LLM quantization behavior. Our work demonstrates that outliers are not
inherent to LLMs but are consequences of training strategies, paving the way
for more efficient LLM deployment. The source code and pretrained checkpoints
are available at https://github.com/dmis-lab/Outlier-Safe-Pre-Training.
Outlier-Safe Pre-Training improves large language model quantization performance by preventing extreme activation outliers through innovative training techniques.
Muon optimizer Single-Scale RMSNorm learnable embedding projection outlier formation quantization performance LLM deployment excess kurtosis