LocalMamba: Visual State Space Model with Windowed Selective Scan
Recent advancements in state space models, notably Mamba, have demonstrated
significant progress in modeling long sequences for tasks like language
understanding. Yet, their application in vision tasks has not markedly
surpassed the performance of traditional Convolutional Neural Networks (CNNs)
and Vision Transformers (ViTs). This paper posits that the key to enhancing
Vision Mamba (ViM) lies in optimizing scan directions for sequence modeling.
Traditional ViM approaches, which flatten spatial tokens, overlook the
preservation of local 2D dependencies, thereby elongating the distance between
adjacent tokens. We introduce a novel local scanning strategy that divides
images into distinct windows, effectively capturing local dependencies while
maintaining a global perspective. Additionally, acknowledging the varying
preferences for scan patterns across different network layers, we propose a
dynamic method to independently search for the optimal scan choices for each
layer, substantially improving performance. Extensive experiments across both
plain and hierarchical models underscore our approach's superiority in
effectively capturing image representations. For example, our model
significantly outperforms Vim-Ti by 3.1% on ImageNet with the same 1.5G FLOPs.
Code is available at: https://github.com/hunto/LocalMamba.
65f3bc49f5b6c6b52fc08c29
Tao Huang
65f3bc49f5b6c6b52fc08c2a
Xiaohuan Pei
claimed_verified
2024-04-16T00:02:44.701Z
6318661051edb569f1b4e68b
/avatars/9b98943bc609ca974abf0abff3e68633.svg
TerryPei
user
TerryPei
65f3bc49f5b6c6b52fc08c2b
Shan You
65f3bc49f5b6c6b52fc08c2c
Fei Wang
65f3bc49f5b6c6b52fc08c2d
Chen Qian
65f3bc49f5b6c6b52fc08c2e
Chang Xu
LocalMamba: Visual State Space Model with Windowed Selective Scan
Recent advancements in state space models, notably Mamba, have demonstrated
significant progress in modeling long sequences for tasks like language
understanding. Yet, their application in vision tasks has not markedly
surpassed the performance of traditional Convolutional Neural Networks (CNNs)
and Vision Transformers (ViTs). This paper posits that the key to enhancing
Vision Mamba (ViM) lies in optimizing scan directions for sequence modeling.
Traditional ViM approaches, which flatten spatial tokens, overlook the
preservation of local 2D dependencies, thereby elongating the distance between
adjacent tokens. We introduce a novel local scanning strategy that divides
images into distinct windows, effectively capturing local dependencies while
maintaining a global perspective. Additionally, acknowledging the varying
preferences for scan patterns across different network layers, we propose a
dynamic method to independently search for the optimal scan choices for each
layer, substantially improving performance. Extensive experiments across both
plain and hierarchical models underscore our approach's superiority in
effectively capturing image representations. For example, our model
significantly outperforms Vim-Ti by 3.1% on ImageNet with the same 1.5G FLOPs.
Code is available at: https://github.com/hunto/LocalMamba.
A novel local scanning strategy for Vision Mamba improves vision task performance by optimizing scan directions and maintaining local 2D dependencies across distinct layers.
state space models Mamba Vision Mamba sequence modeling spatial tokens local dependencies scan directions image windows dynamic scanning ImageNet FLOPs