Towards Next-Level Post-Training Quantization of Hyper-Scale
  Transformers
With the increasing complexity of generative AI models, post-training
quantization (PTQ) has emerged as a promising solution for deploying
hyper-scale models on edge devices such as mobile devices and TVs. Existing PTQ
schemes, however, consume considerable time and resources, which could be a
bottleneck in real situations where frequent model updates and multiple
hyper-parameter tunings are required. As a cost-effective alternative, one-shot
PTQ schemes have been proposed. Still, the performance is somewhat limited
because they cannot consider the inter-layer dependency within the attention
module, which is a very important feature of Transformers. In this paper, we
thus propose a novel PTQ algorithm that balances accuracy and efficiency. The
key idea of the proposed algorithm called aespa is to perform quantization
layer-wise for efficiency while considering cross-layer dependency to preserve
the attention score. Through extensive experiments on various language models
and complexity analysis, we demonstrate that aespa is accurate and efficient in
quantizing Transformer models.
65cda204eedafd0c26a15d2a
Junhan Kim
claimed_verified
2024-10-22T08:03:51.680Z
6609dcee4fa3a72a979e0769
/avatars/588fd3a1e9d6d0952cd6cc7f7e2e8a2e.svg
Junhan
user
SShock92
65cda204eedafd0c26a15d2b
Kyungphil Park
65cda204eedafd0c26a15d2c
Chungman Lee
65cda204eedafd0c26a15d2d
Ho-young Kim
65cda204eedafd0c26a15d2e
Joonyoung Kim
65cda204eedafd0c26a15d2f
Yongkweon Jeon
Towards Next-Level Post-Training Quantization of Hyper-Scale
  Transformers
With the increasing complexity of generative AI models, post-training
quantization (PTQ) has emerged as a promising solution for deploying
hyper-scale models on edge devices such as mobile devices and TVs. Existing PTQ
schemes, however, consume considerable time and resources, which could be a
bottleneck in real situations where frequent model updates and multiple
hyper-parameter tunings are required. As a cost-effective alternative, one-shot
PTQ schemes have been proposed. Still, the performance is somewhat limited
because they cannot consider the inter-layer dependency within the attention
module, which is a very important feature of Transformers. In this paper, we
thus propose a novel PTQ algorithm that balances accuracy and efficiency. The
key idea of the proposed algorithm called aespa is to perform quantization
layer-wise for efficiency while considering cross-layer dependency to preserve
the attention score. Through extensive experiments on various language models
and complexity analysis, we demonstrate that aespa is accurate and efficient in
quantizing Transformer models.
A new PTQ algorithm, aespa, efficiently quantizes Transformer models by balancing accuracy and efficiency while accounting for inter-layer dependencies in attention modules.
post-training quantization PTQ one-shot PTQ attention module Transformers quantization layer-wise cross-layer dependency attention score