Quantizable Transformers: Removing Outliers by Helping Attention Heads
  Do Nothing
Transformer models have been widely adopted in various domains over the last
years, and especially large language models have advanced the field of AI
significantly. Due to their size, the capability of these networks has
increased tremendously, but this has come at the cost of a significant increase
in necessary compute. Quantization is one of the most effective ways to reduce
the computational time and memory consumption of neural networks. Many studies
have shown, however, that modern transformer models tend to learn strong
outliers in their activations, making them difficult to quantize. To retain
acceptable performance, the existence of these outliers requires activations to
be in higher bitwidth or the use of different numeric formats, extra
fine-tuning, or other workarounds. We show that strong outliers are related to
very specific behavior of attention heads that try to learn a "no-op" or just a
partial update of the residual. To achieve the exact zeros needed in the
attention matrix for a no-update, the input to the softmax is pushed to be
larger and larger during training, causing outliers in other parts of the
network. Based on these observations, we propose two simple (independent)
modifications to the attention mechanism - clipped softmax and gated attention.
We empirically show that models pre-trained using our methods learn
significantly smaller outliers while maintaining and sometimes even improving
the floating-point task performance. This enables us to quantize transformers
to full INT8 quantization of the activations without any additional effort. We
demonstrate the effectiveness of our methods on both language models (BERT,
OPT) and vision transformers.
6494f16de7ed8fa81c628049
Yelysei Bondarenko
claimed_verified
2023-06-23T12:20:25.113Z
649571182f09f73fc9c71340
/avatars/5b51bbeac91b6a53712e61a30b670e67.svg
Bondarenko
user
Yelysei
6494f16de7ed8fa81c62804a
Markus Nagel
claimed_verified
2023-06-23T12:20:28.930Z
649576b545b2cf0bf02319f1
/avatars/a25b056ce3a354f21a8074d427ccbd2e.svg
Markus Nagel
user
markusnagel
6494f16de7ed8fa81c62804b
Tijmen Blankevoort
Quantizable Transformers: Removing Outliers by Helping Attention Heads
  Do Nothing
Transformer models have been widely adopted in various domains over the last
years, and especially large language models have advanced the field of AI
significantly. Due to their size, the capability of these networks has
increased tremendously, but this has come at the cost of a significant increase
in necessary compute. Quantization is one of the most effective ways to reduce
the computational time and memory consumption of neural networks. Many studies
have shown, however, that modern transformer models tend to learn strong
outliers in their activations, making them difficult to quantize. To retain
acceptable performance, the existence of these outliers requires activations to
be in higher bitwidth or the use of different numeric formats, extra
fine-tuning, or other workarounds. We show that strong outliers are related to
very specific behavior of attention heads that try to learn a "no-op" or just a
partial update of the residual. To achieve the exact zeros needed in the
attention matrix for a no-update, the input to the softmax is pushed to be
larger and larger during training, causing outliers in other parts of the
network. Based on these observations, we propose two simple (independent)
modifications to the attention mechanism - clipped softmax and gated attention.
We empirically show that models pre-trained using our methods learn
significantly smaller outliers while maintaining and sometimes even improving
the floating-point task performance. This enables us to quantize transformers
to full INT8 quantization of the activations without any additional effort. We
demonstrate the effectiveness of our methods on both language models (BERT,
OPT) and vision transformers.
Two modifications to the attention mechanism, clipped softmax and gated attention, reduce outliers in transformer models, enabling full INT8 quantization without additional effort.
Transformer models large language models quantization computational time memory consumption neural networks activations outliers attention heads no-op residual softmax clipped softmax gated attention INT8 quantization BERT OPT vision transformers