Training Dynamics Impact Post-Training Quantization Robustness
While post-training quantization is widely adopted for efficient deployment
of large language models, the mechanisms underlying quantization robustness
remain unclear. We conduct a comprehensive analysis of quantization degradation
across open-source language model training trajectories up to 32B parameters
and 15T training tokens to accurately assess the relationship between training
dynamics and quantization performance. Our key finding is that quantization
errors in large-scale training runs are driven by a complex interplay between
learning rate and other training hyperparameters. Specifically, once learning
rates decay, validation loss and quantization error diverge, largely
independent of training data scale. To investigate interventions on the
training dynamics and identify specific configurations that can modulate
quantization robustness favorably, we train our own models in controlled
experiments up to 100B tokens. Our results challenge the assumption that
increasing dataset scale inherently compromises quantization effectiveness,
demonstrating instead that strategic training hyperparameter interventions can
improve quantization quality at scale.
68e644c3975ac4c405ef224d
Albert Catalan-Tatjer
68e644c3975ac4c405ef224e
Niccolò Ajroldi
68e644c3975ac4c405ef224f
Jonas Geiping
Training Dynamics Impact Post-Training Quantization Robustness
While post-training quantization is widely adopted for efficient deployment
of large language models, the mechanisms underlying quantization robustness
remain unclear. We conduct a comprehensive analysis of quantization degradation
across open-source language model training trajectories up to 32B parameters
and 15T training tokens to accurately assess the relationship between training
dynamics and quantization performance. Our key finding is that quantization
errors in large-scale training runs are driven by a complex interplay between
learning rate and other training hyperparameters. Specifically, once learning
rates decay, validation loss and quantization error diverge, largely
independent of training data scale. To investigate interventions on the
training dynamics and identify specific configurations that can modulate
quantization robustness favorably, we train our own models in controlled
experiments up to 100B tokens. Our results challenge the assumption that
increasing dataset scale inherently compromises quantization effectiveness,
demonstrating instead that strategic training hyperparameter interventions can
improve quantization quality at scale.
Quantization robustness in large language models is influenced by learning rate and other hyperparameters, not dataset scale, as demonstrated through controlled training experiments.
post-training quantization large language models quantization degradation training trajectories quantization performance learning rate training hyperparameters validation loss quantization error dataset scale training dynamics quantization robustness strategic training hyperparameter interventions