DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis
Diffusion models have achieved great success in image generation, with the
backbone evolving from U-Net to Vision Transformers. However, the computational
cost of Transformers is quadratic to the number of tokens, leading to
significant challenges when dealing with high-resolution images. In this work,
we propose Diffusion Mamba (DiM), which combines the efficiency of Mamba, a
sequence model based on State Space Models (SSM), with the expressive power of
diffusion models for efficient high-resolution image synthesis. To address the
challenge that Mamba cannot generalize to 2D signals, we make several
architecture designs including multi-directional scans, learnable padding
tokens at the end of each row and column, and lightweight local feature
enhancement. Our DiM architecture achieves inference-time efficiency for
high-resolution images. In addition, to further improve training efficiency for
high-resolution image generation with DiM, we investigate ``weak-to-strong''
training strategy that pretrains DiM on low-resolution images (256times 256)
and then finetune it on high-resolution images (512 times 512). We further
explore training-free upsampling strategies to enable the model to generate
higher-resolution images (e.g., 1024times 1024 and 1536times 1536)
without further fine-tuning. Experiments demonstrate the effectiveness and
efficiency of our DiM.
665010198dba23158e9270da
Yao Teng
claimed_verified
2024-10-07T07:31:30.164Z
6427e08288215cee63b1c44d
https://cdn-avatars.huggingface.co/v1/production/uploads/6427e08288215cee63b1c44d/rzaG978FF-ywzicWNl_xl.jpeg
yao teng
user
tytyt
665010198dba23158e9270db
Yue Wu
665010198dba23158e9270dc
Han Shi
claimed_verified
2024-10-13T20:19:32.675Z
637751ad7169ed981703d26a
/avatars/9823f115040a3d6b0240038010e46cb6.svg
Han Shi
user
shihan96
665010198dba23158e9270dd
Xuefei Ning
claimed_verified
2024-06-14T11:47:13.466Z
641031b1a78453b8d96b8420
https://cdn-avatars.huggingface.co/v1/production/uploads/1678782881444-noauth.jpeg
Xuefei Ning
user
Foxfi
665010198dba23158e9270de
Guohao Dai
665010198dba23158e9270df
Yu Wang
665010198dba23158e9270e0
Zhenguo Li
admin_assigned
2024-05-24T11:31:48.034Z
64f6a67f10a01dc75c6fba70
/avatars/47989ccce94245bdb1a18722842ce878.svg
Zhenguo Lee
user
Zhenguo7
665010198dba23158e9270e1
Xihui Liu
admin_assigned
2024-05-24T11:31:27.425Z
65d5ec74cd05bc1eaa125040
/avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
Xihui Liu
user
XihuiLiu
DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis
Diffusion models have achieved great success in image generation, with the
backbone evolving from U-Net to Vision Transformers. However, the computational
cost of Transformers is quadratic to the number of tokens, leading to
significant challenges when dealing with high-resolution images. In this work,
we propose Diffusion Mamba (DiM), which combines the efficiency of Mamba, a
sequence model based on State Space Models (SSM), with the expressive power of
diffusion models for efficient high-resolution image synthesis. To address the
challenge that Mamba cannot generalize to 2D signals, we make several
architecture designs including multi-directional scans, learnable padding
tokens at the end of each row and column, and lightweight local feature
enhancement. Our DiM architecture achieves inference-time efficiency for
high-resolution images. In addition, to further improve training efficiency for
high-resolution image generation with DiM, we investigate ``weak-to-strong''
training strategy that pretrains DiM on low-resolution images (256times 256)
and then finetune it on high-resolution images (512 times 512). We further
explore training-free upsampling strategies to enable the model to generate
higher-resolution images (e.g., 1024times 1024 and 1536times 1536)
without further fine-tuning. Experiments demonstrate the effectiveness and
efficiency of our DiM.
Diffusion Mamba combines the efficiency of sequence models with diffusion models to achieve high-resolution image synthesis at lower computational cost.
diffusion models U-Net Vision Transformers State Space Models multi-directional scans learnable padding tokens lightweight local feature enhancement weak-to-strong training training-free upsampling