MambaMixer: Efficient Selective State Space Models with Dual Token and
  Channel Selection
Recent advances in deep learning have mainly relied on Transformers due to
their data dependency and ability to learn at scale. The attention module in
these architectures, however, exhibits quadratic time and space in input size,
limiting their scalability for long-sequence modeling. Despite recent attempts
to design efficient and effective architecture backbone for multi-dimensional
data, such as images and multivariate time series, existing models are either
data independent, or fail to allow inter- and intra-dimension communication.
Recently, State Space Models (SSMs), and more specifically Selective State
Space Models, with efficient hardware-aware implementation, have shown
promising potential for long sequence modeling. Motivated by the success of
SSMs, we present MambaMixer, a new architecture with data-dependent weights
that uses a dual selection mechanism across tokens and channels, called
Selective Token and Channel Mixer. MambaMixer connects selective mixers using a
weighted averaging mechanism, allowing layers to have direct access to early
features. As a proof of concept, we design Vision MambaMixer (ViM2) and Time
Series MambaMixer (TSM2) architectures based on the MambaMixer block and
explore their performance in various vision and time series forecasting tasks.
Our results underline the importance of selective mixing across both tokens and
channels. In ImageNet classification, object detection, and semantic
segmentation tasks, ViM2 achieves competitive performance with well-established
vision models and outperforms SSM-based vision models. In time series
forecasting, TSM2 achieves outstanding performance compared to state-of-the-art
methods while demonstrating significantly improved computational cost. These
results show that while Transformers, cross-channel attention, and MLPs are
sufficient for good performance in time series forecasting, neither is
necessary.
660a407c4fa3a72a97b28d45
Ali Behrouz
claimed_verified
2024-04-02T11:08:04.279Z
65cccd5134a5d74cbaa9446c
/avatars/5255b734628992106598eae4f2c5848f.svg
Ali Behrouz
user
AliBehrouz
660a407c4fa3a72a97b28d46
Michele Santacatterina
660a407c4fa3a72a97b28d47
Ramin Zabih
MambaMixer: Efficient Selective State Space Models with Dual Token and
  Channel Selection
Recent advances in deep learning have mainly relied on Transformers due to
their data dependency and ability to learn at scale. The attention module in
these architectures, however, exhibits quadratic time and space in input size,
limiting their scalability for long-sequence modeling. Despite recent attempts
to design efficient and effective architecture backbone for multi-dimensional
data, such as images and multivariate time series, existing models are either
data independent, or fail to allow inter- and intra-dimension communication.
Recently, State Space Models (SSMs), and more specifically Selective State
Space Models, with efficient hardware-aware implementation, have shown
promising potential for long sequence modeling. Motivated by the success of
SSMs, we present MambaMixer, a new architecture with data-dependent weights
that uses a dual selection mechanism across tokens and channels, called
Selective Token and Channel Mixer. MambaMixer connects selective mixers using a
weighted averaging mechanism, allowing layers to have direct access to early
features. As a proof of concept, we design Vision MambaMixer (ViM2) and Time
Series MambaMixer (TSM2) architectures based on the MambaMixer block and
explore their performance in various vision and time series forecasting tasks.
Our results underline the importance of selective mixing across both tokens and
channels. In ImageNet classification, object detection, and semantic
segmentation tasks, ViM2 achieves competitive performance with well-established
vision models and outperforms SSM-based vision models. In time series
forecasting, TSM2 achieves outstanding performance compared to state-of-the-art
methods while demonstrating significantly improved computational cost. These
results show that while Transformers, cross-channel attention, and MLPs are
sufficient for good performance in time series forecasting, neither is
necessary.
MambaMixer, a new architecture with selective token and channel mixing, demonstrates competitive performance in vision and outstanding performance in time series forecasting compared to state-of-the-art methods.
Transformers attention module quadratic time quadratic space long-sequence modeling State Space Models Selective State Space Models efficient hardware-aware implementation MambaMixer data-dependent weights Selective Token and Channel Mixer weighted averaging mechanism Vision MambaMixer Time Series MambaMixer ImageNet classification object detection semantic segmentation time series forecasting cross-channel attention MLPs