SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate
  Time series
Transformers have widely adopted attention networks for sequence mixing and
MLPs for channel mixing, playing a pivotal role in achieving breakthroughs
across domains. However, recent literature highlights issues with attention
networks, including low inductive bias and quadratic complexity concerning
input sequence length. State Space Models (SSMs) like S4 and others (Hippo,
Global Convolutions, liquid S4, LRU, Mega, and Mamba), have emerged to address
the above issues to help handle longer sequence lengths. Mamba, while being the
state-of-the-art SSM, has a stability issue when scaled to large networks for
computer vision datasets. We propose SiMBA, a new architecture that introduces
Einstein FFT (EinFFT) for channel modeling by specific eigenvalue computations
and uses the Mamba block for sequence modeling. Extensive performance studies
across image and time-series benchmarks demonstrate that SiMBA outperforms
existing SSMs, bridging the performance gap with state-of-the-art transformers.
Notably, SiMBA establishes itself as the new state-of-the-art SSM on ImageNet
and transfer learning benchmarks such as Stanford Car and Flower as well as
task learning benchmarks as well as seven time series benchmark datasets. The
project page is available on this website
~https://github.com/badripatro/Simba.
6600d54e4d7ed9ef948b1277
Badri N. Patro
admin_assigned
2024-03-25T10:50:15.761Z
64abec030ba6725d692bcd0f
/avatars/8a5cc786de279726bdf6b194f21ac3cb.svg
Badri Narayana Patro
user
badripatro
6600d54e4d7ed9ef948b1278
Vijay S. Agneeswaran
SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate
  Time series
Transformers have widely adopted attention networks for sequence mixing and
MLPs for channel mixing, playing a pivotal role in achieving breakthroughs
across domains. However, recent literature highlights issues with attention
networks, including low inductive bias and quadratic complexity concerning
input sequence length. State Space Models (SSMs) like S4 and others (Hippo,
Global Convolutions, liquid S4, LRU, Mega, and Mamba), have emerged to address
the above issues to help handle longer sequence lengths. Mamba, while being the
state-of-the-art SSM, has a stability issue when scaled to large networks for
computer vision datasets. We propose SiMBA, a new architecture that introduces
Einstein FFT (EinFFT) for channel modeling by specific eigenvalue computations
and uses the Mamba block for sequence modeling. Extensive performance studies
across image and time-series benchmarks demonstrate that SiMBA outperforms
existing SSMs, bridging the performance gap with state-of-the-art transformers.
Notably, SiMBA establishes itself as the new state-of-the-art SSM on ImageNet
and transfer learning benchmarks such as Stanford Car and Flower as well as
task learning benchmarks as well as seven time series benchmark datasets. The
project page is available on this website
~https://github.com/badripatro/Simba.
SiMBA, a new state-space model architecture using Einstein FFT for channel modeling and Mamba blocks for sequence modeling, outperforms existing SSMs and transformers across image and time-series benchmarks.
transformers attention networks MLPs State Space Models S4 Hippo Global Convolutions liquid S4 LRU Mega Mamba Einstein FFT EinFFT channel modeling sequence modeling SiMBA ImageNet Stanford Car Flower task learning time series benchmark datasets