SLAB: Efficient Transformers with Simplified Linear Attention and
  Progressive Re-parameterized Batch Normalization
Transformers have become foundational architectures for both natural language
and computer vision tasks. However, the high computational cost makes it quite
challenging to deploy on resource-constraint devices. This paper investigates
the computational bottleneck modules of efficient transformer, i.e.,
normalization layers and attention modules. LayerNorm is commonly used in
transformer architectures but is not computational friendly due to statistic
calculation during inference. However, replacing LayerNorm with more efficient
BatchNorm in transformer often leads to inferior performance and collapse in
training. To address this problem, we propose a novel method named PRepBN to
progressively replace LayerNorm with re-parameterized BatchNorm in training.
Moreover, we propose a simplified linear attention (SLA) module that is simple
yet effective to achieve strong performance. Extensive experiments on image
classification as well as object detection demonstrate the effectiveness of our
proposed method. For example, our SLAB-Swin obtains 83.6% top-1 accuracy on
ImageNet-1K with 16.2ms latency, which is 2.4ms less than that of
Flatten-Swin with 0.1% higher accuracy. We also evaluated our method for
language modeling task and obtain comparable performance and lower
latency.Codes are publicly available at https://github.com/xinghaochen/SLAB and
https://github.com/mindspore-lab/models/tree/master/research/huawei-noah/SLAB.
664c0e14e60e8ad392878afd
Jialong Guo
admin_assigned
2024-05-21T09:13:53.812Z
6621d02fc50869c1e91522ef
/avatars/db7e40095cbb694e5df148bd2cddad37.svg
guojialong
user
guojialong
664c0e14e60e8ad392878afe
Xinghao Chen
extracted_pending
2024-05-21T02:59:33.252Z
65878c774f9d2b955e025019
/avatars/c039c29e131a8cba96b38048da94971b.svg
Xinghao Chen
user
xinghaochen
664c0e14e60e8ad392878aff
Yehui Tang
admin_assigned
2024-05-21T09:14:01.241Z
64d5deb154bb9eb704f83122
/avatars/86ce09bcca903319051e2307581a43f4.svg
Yehui Tang
user
tangyehui
664c0e14e60e8ad392878b00
Yunhe Wang
admin_assigned
2024-05-21T09:14:07.049Z
658bdf7b925aadd43304f05c
/avatars/64d9e9dea27c376c3bc7b2a54efc2a46.svg
Yunhe Wang
user
MightyCrane
SLAB: Efficient Transformers with Simplified Linear Attention and
  Progressive Re-parameterized Batch Normalization
Transformers have become foundational architectures for both natural language
and computer vision tasks. However, the high computational cost makes it quite
challenging to deploy on resource-constraint devices. This paper investigates
the computational bottleneck modules of efficient transformer, i.e.,
normalization layers and attention modules. LayerNorm is commonly used in
transformer architectures but is not computational friendly due to statistic
calculation during inference. However, replacing LayerNorm with more efficient
BatchNorm in transformer often leads to inferior performance and collapse in
training. To address this problem, we propose a novel method named PRepBN to
progressively replace LayerNorm with re-parameterized BatchNorm in training.
Moreover, we propose a simplified linear attention (SLA) module that is simple
yet effective to achieve strong performance. Extensive experiments on image
classification as well as object detection demonstrate the effectiveness of our
proposed method. For example, our SLAB-Swin obtains 83.6% top-1 accuracy on
ImageNet-1K with 16.2ms latency, which is 2.4ms less than that of
Flatten-Swin with 0.1% higher accuracy. We also evaluated our method for
language modeling task and obtain comparable performance and lower
latency.Codes are publicly available at https://github.com/xinghaochen/SLAB and
https://github.com/mindspore-lab/models/tree/master/research/huawei-noah/SLAB.
New methods improve transformer efficiency by progressively replacing LayerNorm with re-parameterized BatchNorm and using a simplified linear attention module, achieving strong performance with lower latency across image classification and language modeling tasks.
transformers normalization layers attention modules LayerNorm BatchNorm PRepBN re-parameterized BatchNorm simplified linear attention image classification object detection ImageNet-1K Flattened-Swin language modeling