T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with
  Trajectory Stitching
Sampling from diffusion probabilistic models (DPMs) is often expensive for
high-quality image generation and typically requires many steps with a large
model. In this paper, we introduce sampling Trajectory Stitching T-Stitch, a
simple yet efficient technique to improve the sampling efficiency with little
or no generation degradation. Instead of solely using a large DPM for the
entire sampling trajectory, T-Stitch first leverages a smaller DPM in the
initial steps as a cheap drop-in replacement of the larger DPM and switches to
the larger DPM at a later stage. Our key insight is that different diffusion
models learn similar encodings under the same training data distribution and
smaller models are capable of generating good global structures in the early
steps. Extensive experiments demonstrate that T-Stitch is training-free,
generally applicable for different architectures, and complements most existing
fast sampling techniques with flexible speed and quality trade-offs. On DiT-XL,
for example, 40% of the early timesteps can be safely replaced with a 10x
faster DiT-S without performance drop on class-conditional ImageNet generation.
We further show that our method can also be used as a drop-in technique to not
only accelerate the popular pretrained stable diffusion (SD) models but also
improve the prompt alignment of stylized SD models from the public model zoo.
Code is released at https://github.com/NVlabs/T-Stitch
65d807403381a5be2777e499
Zizheng Pan
admin_assigned
2024-02-23T12:59:11.640Z
629d55b53a3221bb2117422c
/avatars/e39f25cd7190d282478cdac20a7700ee.svg
Zizheng Pan
user
zizhpan
65d807403381a5be2777e49a
Bohan Zhuang
extracted_confirmed
2024-02-23T09:07:36.541Z
65a8a828ef14f9e6037dfad2
/avatars/55fdc0393b2e37ac998ddbdb1e1ff636.svg
Bohan Zhuang
user
BohanZ
65d807403381a5be2777e49b
De-An Huang
admin_assigned
2024-02-23T12:59:19.059Z
641d1c5ec3983aa94915c162
/avatars/127985b837ecf61e43c835deee578b5e.svg
De-An Huang
user
deahuang
65d807403381a5be2777e49c
Weili Nie
admin_assigned
2024-02-23T12:59:27.287Z
64c1a69e226e016da8450ae2
/avatars/54c161e8b8543244ed13cbe47017624e.svg
Weili Nie
user
xiaoli08
65d807403381a5be2777e49d
Zhiding Yu
claimed_verified
2024-08-29T07:25:21.964Z
66c8037c737ba92ae3fe0322
https://cdn-avatars.huggingface.co/v1/production/uploads/66c8037c737ba92ae3fe0322/WR_Yh5DWOVVh7IFlF24NM.jpeg
Zhiding Yu
user
Zhiding
65d807403381a5be2777e49e
Chaowei Xiao
65d807403381a5be2777e49f
Jianfei Cai
65d807403381a5be2777e4a0
Anima Anandkumar
admin_assigned
2024-02-23T12:59:58.141Z
6532920b3e385cfc6002938d
/avatars/cb9cc6d2733031582c83f56dc6cd1dd5.svg
Anima Anandkumar
user
animakumar
T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with
  Trajectory Stitching
Sampling from diffusion probabilistic models (DPMs) is often expensive for
high-quality image generation and typically requires many steps with a large
model. In this paper, we introduce sampling Trajectory Stitching T-Stitch, a
simple yet efficient technique to improve the sampling efficiency with little
or no generation degradation. Instead of solely using a large DPM for the
entire sampling trajectory, T-Stitch first leverages a smaller DPM in the
initial steps as a cheap drop-in replacement of the larger DPM and switches to
the larger DPM at a later stage. Our key insight is that different diffusion
models learn similar encodings under the same training data distribution and
smaller models are capable of generating good global structures in the early
steps. Extensive experiments demonstrate that T-Stitch is training-free,
generally applicable for different architectures, and complements most existing
fast sampling techniques with flexible speed and quality trade-offs. On DiT-XL,
for example, 40% of the early timesteps can be safely replaced with a 10x
faster DiT-S without performance drop on class-conditional ImageNet generation.
We further show that our method can also be used as a drop-in technique to not
only accelerate the popular pretrained stable diffusion (SD) models but also
improve the prompt alignment of stylized SD models from the public model zoo.
Code is released at https://github.com/NVlabs/T-Stitch
T-Stitch, a technique for efficient sampling from diffusion probabilistic models, leverages smaller models early in the process to reduce computational cost without degrading image quality.
diffusion probabilistic models sampling Trajectory Stitching T-Stitch sampling efficiency diffusion models training-free fast sampling techniques DiT-XL DiT-S pretrained stable diffusion prompt alignment