Training Transformers with 4-bit Integers
Quantizing the activation, weight, and gradient to 4-bit is promising to
accelerate neural network training. However, existing 4-bit training methods
require custom numerical formats which are not supported by contemporary
hardware. In this work, we propose a training method for transformers with all
matrix multiplications implemented with the INT4 arithmetic. Training with an
ultra-low INT4 precision is challenging. To achieve this, we carefully analyze
the specific structures of activation and gradients in transformers to propose
dedicated quantizers for them. For forward propagation, we identify the
challenge of outliers and propose a Hadamard quantizer to suppress the
outliers. For backpropagation, we leverage the structural sparsity of gradients
by proposing bit splitting and leverage score sampling techniques to quantize
gradients accurately. Our algorithm achieves competitive accuracy on a wide
range of tasks including natural language understanding, machine translation,
and image classification. Unlike previous 4-bit training methods, our algorithm
can be implemented on the current generation of GPUs. Our prototypical linear
operator implementation is up to 2.2 times faster than the FP16 counterparts
and speeds up the training by up to 35.1%.
6493c88bbc32063c9e46929b
Haocheng Xi
claimed_verified
2023-06-23T07:31:51.474Z
6452a2cbc895e6437dc567e2
/avatars/f488ebe2fbece0e8ef4e9d6c95898f3a.svg
Haocheng Xi
user
Xihc20
6493c88bbc32063c9e46929c
Changhao Li
claimed_verified
2025-07-23T08:39:42.875Z
660026d9573abbcdb975aa13
https://cdn-avatars.huggingface.co/v1/production/uploads/660026d9573abbcdb975aa13/XktfFxTMC_d1xJ4pDJLum.jpeg
Changhao
user
lichangh20
6493c88bbc32063c9e46929d
Jianfei Chen
6493c88bbc32063c9e46929e
Jun Zhu
Training Transformers with 4-bit Integers
Quantizing the activation, weight, and gradient to 4-bit is promising to
accelerate neural network training. However, existing 4-bit training methods
require custom numerical formats which are not supported by contemporary
hardware. In this work, we propose a training method for transformers with all
matrix multiplications implemented with the INT4 arithmetic. Training with an
ultra-low INT4 precision is challenging. To achieve this, we carefully analyze
the specific structures of activation and gradients in transformers to propose
dedicated quantizers for them. For forward propagation, we identify the
challenge of outliers and propose a Hadamard quantizer to suppress the
outliers. For backpropagation, we leverage the structural sparsity of gradients
by proposing bit splitting and leverage score sampling techniques to quantize
gradients accurately. Our algorithm achieves competitive accuracy on a wide
range of tasks including natural language understanding, machine translation,
and image classification. Unlike previous 4-bit training methods, our algorithm
can be implemented on the current generation of GPUs. Our prototypical linear
operator implementation is up to 2.2 times faster than the FP16 counterparts
and speeds up the training by up to 35.1%.
A novel method for training transformers with 4-bit quantization achieves competitive accuracy and accelerates training on current GPUs.
activation quantization weight quantization gradient quantization INT4 arithmetic Hadamard quantizer bit splitting score sampling structural sparsity natural language understanding machine translation image classification FP16 counterparts