Make a Cheap Scaling: A Self-Cascade Diffusion Model for
  Higher-Resolution Adaptation
Diffusion models have proven to be highly effective in image and video
generation; however, they still face composition challenges when generating
images of varying sizes due to single-scale training data. Adapting large
pre-trained diffusion models for higher resolution demands substantial
computational and optimization resources, yet achieving a generation capability
comparable to low-resolution models remains elusive. This paper proposes a
novel self-cascade diffusion model that leverages the rich knowledge gained
from a well-trained low-resolution model for rapid adaptation to
higher-resolution image and video generation, employing either tuning-free or
cheap upsampler tuning paradigms. Integrating a sequence of multi-scale
upsampler modules, the self-cascade diffusion model can efficiently adapt to a
higher resolution, preserving the original composition and generation
capabilities. We further propose a pivot-guided noise re-schedule strategy to
speed up the inference process and improve local structural details. Compared
to full fine-tuning, our approach achieves a 5X training speed-up and requires
only an additional 0.002M tuning parameters. Extensive experiments demonstrate
that our approach can quickly adapt to higher resolution image and video
synthesis by fine-tuning for just 10k steps, with virtually no additional
inference time.
65d2c60b71ca8e594f7a0eac
Lanqing Guo
65d2c60b71ca8e594f7a0ead
Yingqing He
admin_assigned
2024-02-19T11:29:50.604Z
630231bf7e137e3d6b3b0645
/avatars/dad52f8955110f0a2caeb613d6aa3ea2.svg
He
user
Yingqing
65d2c60b71ca8e594f7a0eae
Haoxin Chen
admin_assigned
2024-02-19T11:30:19.045Z
6305cb2f435ec751b723d80a
/avatars/5e8519d9814038726bf98f4104f74568.svg
Haoxin Chen
user
paulai
65d2c60b71ca8e594f7a0eaf
Menghan Xia
admin_assigned
2024-02-19T11:30:36.830Z
63401c89f81b9d101361f712
https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
Richard
user
menghanxia
65d2c60b71ca8e594f7a0eb0
Xiaodong Cun
admin_assigned
2024-02-19T11:30:49.206Z
63184c517ca1b876d99b7e0e
https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
Xiaodong Cun
user
vinthony
65d2c60b71ca8e594f7a0eb1
Yufei Wang
admin_assigned
2024-02-19T11:31:55.148Z
64a48f758b961fd6fa385737
/avatars/328fdf1a4dca5f2bafbedf1d4af83f7c.svg
Yufei Wang
user
lomo-1
65d2c60b71ca8e594f7a0eb2
Siyu Huang
admin_assigned
2024-02-19T11:32:02.611Z
647060e37fd7ecdbd0e982cd
/avatars/9bb1b61182f837e45722931695c0b8d0.svg
siyu huang
user
hsyyy
65d2c60b71ca8e594f7a0eb3
Yong Zhang
65d2c60b71ca8e594f7a0eb4
Xintao Wang
admin_assigned
2024-02-19T11:32:41.481Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
65d2c60b71ca8e594f7a0eb5
Qifeng Chen
admin_assigned
2024-02-19T11:32:48.176Z
6467b121e7a6a374fd19b44b
/avatars/3f2874d58986d651aef55e3408b05700.svg
Qifeng Chen
user
cqf
65d2c60b71ca8e594f7a0eb6
Ying Shan
admin_assigned
2024-02-19T11:33:09.709Z
63ca3ddc04c979828310bfcb
/avatars/615e0d8622950b4408b40d550f02a894.svg
Ying Shan
user
yshan2u
65d2c60b71ca8e594f7a0eb7
Bihan Wen
Make a Cheap Scaling: A Self-Cascade Diffusion Model for
  Higher-Resolution Adaptation
Diffusion models have proven to be highly effective in image and video
generation; however, they still face composition challenges when generating
images of varying sizes due to single-scale training data. Adapting large
pre-trained diffusion models for higher resolution demands substantial
computational and optimization resources, yet achieving a generation capability
comparable to low-resolution models remains elusive. This paper proposes a
novel self-cascade diffusion model that leverages the rich knowledge gained
from a well-trained low-resolution model for rapid adaptation to
higher-resolution image and video generation, employing either tuning-free or
cheap upsampler tuning paradigms. Integrating a sequence of multi-scale
upsampler modules, the self-cascade diffusion model can efficiently adapt to a
higher resolution, preserving the original composition and generation
capabilities. We further propose a pivot-guided noise re-schedule strategy to
speed up the inference process and improve local structural details. Compared
to full fine-tuning, our approach achieves a 5X training speed-up and requires
only an additional 0.002M tuning parameters. Extensive experiments demonstrate
that our approach can quickly adapt to higher resolution image and video
synthesis by fine-tuning for just 10k steps, with virtually no additional
inference time.
A self-cascade diffusion model that uses multi-scale upsampling and a pivot-guided noise re-schedule strategy efficiently adapts high-resolution image and video generation from a low-resolution diffusion model.
diffusion models self-cascade diffusion model multi-scale upsampler pivot-guided noise re-schedule strategy high-resolution low-resolution image generation video generation training speed-up fine-tuning