Vision Mamba: Efficient Visual Representation Learning with
  Bidirectional State Space Model
Recently the state space models (SSMs) with efficient hardware-aware designs,
i.e., Mamba, have shown great potential for long sequence modeling. Building
efficient and generic vision backbones purely upon SSMs is an appealing
direction. However, representing visual data is challenging for SSMs due to the
position-sensitivity of visual data and the requirement of global context for
visual understanding. In this paper, we show that the reliance of visual
representation learning on self-attention is not necessary and propose a new
generic vision backbone with bidirectional Mamba blocks (Vim), which marks the
image sequences with position embeddings and compresses the visual
representation with bidirectional state space models. On ImageNet
classification, COCO object detection, and ADE20k semantic segmentation tasks,
Vim achieves higher performance compared to well-established vision
transformers like DeiT, while also demonstrating significantly improved
computation & memory efficiency. For example, Vim is 2.8times faster than
DeiT and saves 86.8% GPU memory when performing batch inference to extract
features on images with a resolution of 1248times1248. The results
demonstrate that Vim is capable of overcoming the computation & memory
constraints on performing Transformer-style understanding for high-resolution
images and it has great potential to become the next-generation backbone for
vision foundation models. Code is available at https://github.com/hustvl/Vim.
65a8aa9e87dd2bd7f900d536
Lianghui Zhu
admin_assigned
2024-01-18T09:03:35.982Z
643632c6a4bd75c62cc2d3b4
https://cdn-avatars.huggingface.co/v1/production/uploads/643632c6a4bd75c62cc2d3b4/rCqEb3_w54-QH2QSP2JhM.jpeg
Lianghui Zhu
user
LianghuiZhu
65a8aa9e87dd2bd7f900d537
Bencheng Liao
claimed_verified
2024-11-27T22:21:19.701Z
6577073fc2bf55b1f6bafb49
/avatars/58803398b1a918b7570db17893e65122.svg
Bencheng liao
user
LegendBC
65a8aa9e87dd2bd7f900d538
Qian Zhang
65a8aa9e87dd2bd7f900d539
Xinlong Wang
admin_assigned
2024-01-18T09:04:16.206Z
63ca558304c979828311c5a5
/avatars/2a439d79fba2f987cabe780d10c94d25.svg
Xinlong Wang
user
xinlongwang
65a8aa9e87dd2bd7f900d53a
Wenyu Liu
65a8aa9e87dd2bd7f900d53b
Xinggang Wang
admin_assigned
2024-01-18T09:04:50.536Z
62600de6d47e3dbae32ce1ce
/avatars/a536417cfec6e10ac415091bd1829426.svg
Xinggang Wang
user
xinggangw
Vision Mamba: Efficient Visual Representation Learning with
  Bidirectional State Space Model
Recently the state space models (SSMs) with efficient hardware-aware designs,
i.e., Mamba, have shown great potential for long sequence modeling. Building
efficient and generic vision backbones purely upon SSMs is an appealing
direction. However, representing visual data is challenging for SSMs due to the
position-sensitivity of visual data and the requirement of global context for
visual understanding. In this paper, we show that the reliance of visual
representation learning on self-attention is not necessary and propose a new
generic vision backbone with bidirectional Mamba blocks (Vim), which marks the
image sequences with position embeddings and compresses the visual
representation with bidirectional state space models. On ImageNet
classification, COCO object detection, and ADE20k semantic segmentation tasks,
Vim achieves higher performance compared to well-established vision
transformers like DeiT, while also demonstrating significantly improved
computation & memory efficiency. For example, Vim is 2.8times faster than
DeiT and saves 86.8% GPU memory when performing batch inference to extract
features on images with a resolution of 1248times1248. The results
demonstrate that Vim is capable of overcoming the computation & memory
constraints on performing Transformer-style understanding for high-resolution
images and it has great potential to become the next-generation backbone for
vision foundation models. Code is available at https://github.com/hustvl/Vim.
A new vision backbone using bidirectional Mamba blocks (Vim) outperforms established vision transformers like DeiT in terms of performance and efficiency on high-resolution image tasks.
state space models SSMs Mamba bidirectional Mamba blocks Vim position embeddings bidirectional state space models vision transformers DeiT ImageNet COCO object detection ADE20k semantic segmentation