EfficientViT: Memory Efficient Vision Transformer with Cascaded Group
  Attention
Vision transformers have shown great success due to their high model
capabilities. However, their remarkable performance is accompanied by heavy
computation costs, which makes them unsuitable for real-time applications. In
this paper, we propose a family of high-speed vision transformers named
EfficientViT. We find that the speed of existing transformer models is commonly
bounded by memory inefficient operations, especially the tensor reshaping and
element-wise functions in MHSA. Therefore, we design a new building block with
a sandwich layout, i.e., using a single memory-bound MHSA between efficient FFN
layers, which improves memory efficiency while enhancing channel communication.
Moreover, we discover that the attention maps share high similarities across
heads, leading to computational redundancy. To address this, we present a
cascaded group attention module feeding attention heads with different splits
of the full feature, which not only saves computation cost but also improves
attention diversity. Comprehensive experiments demonstrate EfficientViT
outperforms existing efficient models, striking a good trade-off between speed
and accuracy. For instance, our EfficientViT-M5 surpasses MobileNetV3-Large by
1.9% in accuracy, while getting 40.4% and 45.2% higher throughput on Nvidia
V100 GPU and Intel Xeon CPU, respectively. Compared to the recent efficient
model MobileViT-XXS, EfficientViT-M2 achieves 1.8% superior accuracy, while
running 5.8x/3.7x faster on the GPU/CPU, and 7.4x faster when converted to ONNX
format. Code and models are available at
https://github.com/microsoft/Cream/tree/main/EfficientViT.
645dc520ce72244df7b52bc3
Xinyu Liu
645dc520ce72244df7b52bc4
Houwen Peng
admin_assigned
2023-06-02T12:57:40.098Z
631952b6f18d0b5d999ca397
/avatars/eb87dfc142a7602f8fb888f7b7b60d38.svg
Peng
user
Houwen
645dc520ce72244df7b52bc5
Ningxin Zheng
645dc520ce72244df7b52bc6
Yuqing Yang
645dc520ce72244df7b52bc7
Han Hu
645dc520ce72244df7b52bc8
Yixuan Yuan
admin_assigned
2023-06-02T12:59:27.708Z
640fdfc9f2d7c41a1ea112ef
/avatars/780328c388ac4bc9acdf063e7833259d.svg
yxyuan
user
yixuanyuan
EfficientViT: Memory Efficient Vision Transformer with Cascaded Group
  Attention
Vision transformers have shown great success due to their high model
capabilities. However, their remarkable performance is accompanied by heavy
computation costs, which makes them unsuitable for real-time applications. In
this paper, we propose a family of high-speed vision transformers named
EfficientViT. We find that the speed of existing transformer models is commonly
bounded by memory inefficient operations, especially the tensor reshaping and
element-wise functions in MHSA. Therefore, we design a new building block with
a sandwich layout, i.e., using a single memory-bound MHSA between efficient FFN
layers, which improves memory efficiency while enhancing channel communication.
Moreover, we discover that the attention maps share high similarities across
heads, leading to computational redundancy. To address this, we present a
cascaded group attention module feeding attention heads with different splits
of the full feature, which not only saves computation cost but also improves
attention diversity. Comprehensive experiments demonstrate EfficientViT
outperforms existing efficient models, striking a good trade-off between speed
and accuracy. For instance, our EfficientViT-M5 surpasses MobileNetV3-Large by
1.9% in accuracy, while getting 40.4% and 45.2% higher throughput on Nvidia
V100 GPU and Intel Xeon CPU, respectively. Compared to the recent efficient
model MobileViT-XXS, EfficientViT-M2 achieves 1.8% superior accuracy, while
running 5.8x/3.7x faster on the GPU/CPU, and 7.4x faster when converted to ONNX
format. Code and models are available at
https://github.com/microsoft/Cream/tree/main/EfficientViT.
EfficientViT addresses memory inefficiency and computational redundancy in existing vision transformers, offering improved speed and accuracy.
vision transformers EfficientViT memory-bound operations tensor reshaping element-wise functions MHSA sandwich layout efficient FFN layers channel communication attention maps computational redundancy cascaded group attention module feature splits attention diversity throughput MobileNetV3-Large MobileViT-XXS ONNX format