VSSD: Vision Mamba with Non-Casual State Space Duality
Vision transformers have significantly advanced the field of computer vision,
offering robust modeling capabilities and global receptive field. However,
their high computational demands limit their applicability in processing long
sequences. To tackle this issue, State Space Models (SSMs) have gained
prominence in vision tasks as they offer linear computational complexity.
Recently, State Space Duality (SSD), an improved variant of SSMs, was
introduced in Mamba2 to enhance model performance and efficiency. However, the
inherent causal nature of SSD/SSMs restricts their applications in non-causal
vision tasks. To address this limitation, we introduce Visual State Space
Duality (VSSD) model, which has a non-causal format of SSD. Specifically, we
propose to discard the magnitude of interactions between the hidden state and
tokens while preserving their relative weights, which relieves the dependencies
of token contribution on previous tokens. Together with the involvement of
multi-scan strategies, we show that the scanning results can be integrated to
achieve non-causality, which not only improves the performance of SSD in vision
tasks but also enhances its efficiency. We conduct extensive experiments on
various benchmarks including image classification, detection, and segmentation,
where VSSD surpasses existing state-of-the-art SSM-based models. Code and
weights are available at https://github.com/YuHengsss/VSSD.
66a6fc1de103701b1e53417f
Yuheng Shi
extracted_pending
2024-07-29T02:19:10.175Z
65d6c252ca16ef9ba7e76664
/avatars/45d6e8bdd7b944e912f6674696a941b3.svg
yuhengshi
user
YuhengSSS
66a6fc1de103701b1e534180
Minjing Dong
admin_assigned
2024-07-29T08:52:58.942Z
645250dc1595002ddc8682d6
/avatars/94d02ae79b2750a87c7e2a12a984531a.svg
Minjing DONG
user
UniSerj
66a6fc1de103701b1e534181
Mingjia Li
66a6fc1de103701b1e534182
Chang Xu
VSSD: Vision Mamba with Non-Casual State Space Duality
Vision transformers have significantly advanced the field of computer vision,
offering robust modeling capabilities and global receptive field. However,
their high computational demands limit their applicability in processing long
sequences. To tackle this issue, State Space Models (SSMs) have gained
prominence in vision tasks as they offer linear computational complexity.
Recently, State Space Duality (SSD), an improved variant of SSMs, was
introduced in Mamba2 to enhance model performance and efficiency. However, the
inherent causal nature of SSD/SSMs restricts their applications in non-causal
vision tasks. To address this limitation, we introduce Visual State Space
Duality (VSSD) model, which has a non-causal format of SSD. Specifically, we
propose to discard the magnitude of interactions between the hidden state and
tokens while preserving their relative weights, which relieves the dependencies
of token contribution on previous tokens. Together with the involvement of
multi-scan strategies, we show that the scanning results can be integrated to
achieve non-causality, which not only improves the performance of SSD in vision
tasks but also enhances its efficiency. We conduct extensive experiments on
various benchmarks including image classification, detection, and segmentation,
where VSSD surpasses existing state-of-the-art SSM-based models. Code and
weights are available at https://github.com/YuHengsss/VSSD.
The Visual State Space Duality (VSSD) model enhances the performance and efficiency of State Space Models in vision tasks by introducing a non-causal format.
Vision transformers State Space Models State Space Duality Visual State Space Duality hidden state tokens non-causal format multi-scan strategies image classification detection segmentation