DualToken-ViT: Position-aware Efficient Vision Transformer with Dual
  Token Fusion
Self-attention-based vision transformers (ViTs) have emerged as a highly
competitive architecture in computer vision. Unlike convolutional neural
networks (CNNs), ViTs are capable of global information sharing. With the
development of various structures of ViTs, ViTs are increasingly advantageous
for many vision tasks. However, the quadratic complexity of self-attention
renders ViTs computationally intensive, and their lack of inductive biases of
locality and translation equivariance demands larger model sizes compared to
CNNs to effectively learn visual features. In this paper, we propose a
light-weight and efficient vision transformer model called DualToken-ViT that
leverages the advantages of CNNs and ViTs. DualToken-ViT effectively fuses the
token with local information obtained by convolution-based structure and the
token with global information obtained by self-attention-based structure to
achieve an efficient attention structure. In addition, we use position-aware
global tokens throughout all stages to enrich the global information, which
further strengthening the effect of DualToken-ViT. Position-aware global tokens
also contain the position information of the image, which makes our model
better for vision tasks. We conducted extensive experiments on image
classification, object detection and semantic segmentation tasks to demonstrate
the effectiveness of DualToken-ViT. On the ImageNet-1K dataset, our models of
different scales achieve accuracies of 75.4% and 79.4% with only 0.5G and 1.0G
FLOPs, respectively, and our model with 1.0G FLOPs outperforms LightViT-T using
global tokens by 0.7%.
65111a8d9d93eec4a773cb6f
Zhenzhen Chu
65111a8d9d93eec4a773cb70
Jiayu Chen
65111a8d9d93eec4a773cb71
Cen Chen
65111a8d9d93eec4a773cb72
Chengyu Wang
admin_assigned
2023-09-25T11:25:49.484Z
62a1ee69e497a54ddb32a998
/avatars/bf6ceaba75f09e9e30cd52a803acfd32.svg
Chengyu Wang
user
chywang
65111a8d9d93eec4a773cb73
Ziheng Wu
admin_assigned
2023-09-25T11:25:21.986Z
63b7bf7fca2f378e7103e182
/avatars/73b5179b888293a7039e9bc42d129c61.svg
wuziheng
user
wuziheng
65111a8d9d93eec4a773cb74
Jun Huang
65111a8d9d93eec4a773cb75
Weining Qian
admin_assigned
2023-09-25T11:24:50.188Z
643916835aa69077ffb27c8a
/avatars/3eeb0b8ea703d3c873839ad0fd062352.svg
Weining Qian
user
wnqian
DualToken-ViT: Position-aware Efficient Vision Transformer with Dual
  Token Fusion
Self-attention-based vision transformers (ViTs) have emerged as a highly
competitive architecture in computer vision. Unlike convolutional neural
networks (CNNs), ViTs are capable of global information sharing. With the
development of various structures of ViTs, ViTs are increasingly advantageous
for many vision tasks. However, the quadratic complexity of self-attention
renders ViTs computationally intensive, and their lack of inductive biases of
locality and translation equivariance demands larger model sizes compared to
CNNs to effectively learn visual features. In this paper, we propose a
light-weight and efficient vision transformer model called DualToken-ViT that
leverages the advantages of CNNs and ViTs. DualToken-ViT effectively fuses the
token with local information obtained by convolution-based structure and the
token with global information obtained by self-attention-based structure to
achieve an efficient attention structure. In addition, we use position-aware
global tokens throughout all stages to enrich the global information, which
further strengthening the effect of DualToken-ViT. Position-aware global tokens
also contain the position information of the image, which makes our model
better for vision tasks. We conducted extensive experiments on image
classification, object detection and semantic segmentation tasks to demonstrate
the effectiveness of DualToken-ViT. On the ImageNet-1K dataset, our models of
different scales achieve accuracies of 75.4% and 79.4% with only 0.5G and 1.0G
FLOPs, respectively, and our model with 1.0G FLOPs outperforms LightViT-T using
global tokens by 0.7%.
DualToken-ViT combines the strengths of CNNs and ViTs to achieve efficient vision processing with enhanced global and position-aware features.
self-attention vision transformers ViTs convolutional neural networks CNNs quadratic complexity global information sharing position-aware global tokens image classification object detection semantic segmentation ImageNet-1K FLOPs