ZigMa: Zigzag Mamba Diffusion Model
The diffusion model has long been plagued by scalability and quadratic
complexity issues, especially within transformer-based structures. In this
study, we aim to leverage the long sequence modeling capability of a
State-Space Model called Mamba to extend its applicability to visual data
generation. Firstly, we identify a critical oversight in most current
Mamba-based vision methods, namely the lack of consideration for spatial
continuity in the scan scheme of Mamba. Secondly, building upon this insight,
we introduce a simple, plug-and-play, zero-parameter method named Zigzag Mamba,
which outperforms Mamba-based baselines and demonstrates improved speed and
memory utilization compared to transformer-based baselines. Lastly, we
integrate Zigzag Mamba with the Stochastic Interpolant framework to investigate
the scalability of the model on large-resolution visual datasets, such as
FacesHQ 1024times 1024 and UCF101, MultiModal-CelebA-HQ, and MS COCO
256times 256. Code will be released at https://taohu.me/zigma/
65fbb705a12d7b72306afaef
Vincent Tao Hu
admin_assigned
2024-03-21T11:36:48.119Z
63dcd7ac22cc06e76a8484ce
https://cdn-avatars.huggingface.co/v1/production/uploads/63dcd7ac22cc06e76a8484ce/4O8NcevQ0ZWhJJ1r4Pbam.jpeg
Vincent Tao Hu
user
taohu
65fbb705a12d7b72306afaf0
Stefan Andreas Baumann
claimed_verified
2024-03-25T09:44:16.012Z
6471c12a0c2b5fdaf1f07c45
/avatars/6783067212d24d1e716a8b4c64df61b4.svg
Stefan Baumann
user
stefan-baumann
65fbb705a12d7b72306afaf1
Ming Gui
claimed_verified
2024-09-30T08:10:06.744Z
6319f120d47392f426cd48be
/avatars/fa6d8761ef7a1a86a8a7e97408dc3142.svg
Ming Gui
user
mgui
65fbb705a12d7b72306afaf2
Olga Grebenkova
claimed_verified
2024-07-08T20:01:20.816Z
633b481d0d68f86e2d96ed84
https://cdn-avatars.huggingface.co/v1/production/uploads/633b481d0d68f86e2d96ed84/ACvGaMbeYSYIGiez0jJrE.jpeg
Olga Grebenkova
user
magistrkoljan
65fbb705a12d7b72306afaf3
Pingchuan Ma
65fbb705a12d7b72306afaf4
Johannes Fischer
claimed_verified
2024-03-22T09:32:46.591Z
63cea5dd257cc48a04f8dc21
/avatars/db697e0954ff5be392b8ab588df2f0c4.svg
J. Schusterbauer
user
joh-schb
65fbb705a12d7b72306afaf5
Bjorn Ommer
ZigMa: Zigzag Mamba Diffusion Model
The diffusion model has long been plagued by scalability and quadratic
complexity issues, especially within transformer-based structures. In this
study, we aim to leverage the long sequence modeling capability of a
State-Space Model called Mamba to extend its applicability to visual data
generation. Firstly, we identify a critical oversight in most current
Mamba-based vision methods, namely the lack of consideration for spatial
continuity in the scan scheme of Mamba. Secondly, building upon this insight,
we introduce a simple, plug-and-play, zero-parameter method named Zigzag Mamba,
which outperforms Mamba-based baselines and demonstrates improved speed and
memory utilization compared to transformer-based baselines. Lastly, we
integrate Zigzag Mamba with the Stochastic Interpolant framework to investigate
the scalability of the model on large-resolution visual datasets, such as
FacesHQ 1024times 1024 and UCF101, MultiModal-CelebA-HQ, and MS COCO
256times 256. Code will be released at https://taohu.me/zigma/
Zigzag Mamba, an improvement over Mamba, enhances visual data generation with better spatial continuity, speed, and memory usage, and is scalable with Stochastic Interpolant on large-resolution datasets.
diffusion model scalability quadratic complexity State-Space Model Mamba spatial continuity scan scheme Zigzag Mamba zero-parameter method Stochastic Interpolant large-resolution visual datasets FacesHQ UCF101 MultiModal-CelebA-HQ MS COCO