Improved Distribution Matching Distillation for Fast Image Synthesis
Recent approaches have shown promises distilling diffusion models into
efficient one-step generators. Among them, Distribution Matching Distillation
(DMD) produces one-step generators that match their teacher in distribution,
without enforcing a one-to-one correspondence with the sampling trajectories of
their teachers. However, to ensure stable training, DMD requires an additional
regression loss computed using a large set of noise-image pairs generated by
the teacher with many steps of a deterministic sampler. This is costly for
large-scale text-to-image synthesis and limits the student's quality, tying it
too closely to the teacher's original sampling paths. We introduce DMD2, a set
of techniques that lift this limitation and improve DMD training. First, we
eliminate the regression loss and the need for expensive dataset construction.
We show that the resulting instability is due to the fake critic not estimating
the distribution of generated samples accurately and propose a two time-scale
update rule as a remedy. Second, we integrate a GAN loss into the distillation
procedure, discriminating between generated samples and real images. This lets
us train the student model on real data, mitigating the imperfect real score
estimation from the teacher model, and enhancing quality. Lastly, we modify the
training procedure to enable multi-step sampling. We identify and address the
training-inference input mismatch problem in this setting, by simulating
inference-time generator samples during training time. Taken together, our
improvements set new benchmarks in one-step image generation, with FID scores
of 1.28 on ImageNet-64x64 and 8.35 on zero-shot COCO 2014, surpassing the
original teacher despite a 500X reduction in inference cost. Further, we show
our approach can generate megapixel images by distilling SDXL, demonstrating
exceptional visual quality among few-step methods.
665016c254eda33d5406cde7
Tianwei Yin
admin_assigned
2024-05-24T11:28:18.431Z
63363b864067f020756275b7
/avatars/bf18f9f76183f81589a80b3d0215f1a5.svg
Tianwei Yin
user
tianweiy
665016c254eda33d5406cde8
Michaël Gharbi
admin_assigned
2024-05-24T11:28:27.247Z
658161ad208369a9403ca9a1
/avatars/e2324ab2a712100681b0b8127305212b.svg
Michael Gharbi
user
mgharbi
665016c254eda33d5406cde9
Taesung Park
admin_assigned
2024-05-24T11:28:57.967Z
637563c59ef4e3b87a60d503
/avatars/8264de115e6305e4720c78fc7dce7f8d.svg
Taesung Park
user
taesungp
665016c254eda33d5406cdea
Richard Zhang
665016c254eda33d5406cdeb
Eli Shechtman
665016c254eda33d5406cdec
Fredo Durand
665016c254eda33d5406cded
William T. Freeman
admin_assigned
2024-05-24T11:29:29.740Z
654e866a85a5e608059a9b4f
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/O0T97EupMHH_iXcv1xYFr.png
William Freeman
user
mrpuppt
Improved Distribution Matching Distillation for Fast Image Synthesis
Recent approaches have shown promises distilling diffusion models into
efficient one-step generators. Among them, Distribution Matching Distillation
(DMD) produces one-step generators that match their teacher in distribution,
without enforcing a one-to-one correspondence with the sampling trajectories of
their teachers. However, to ensure stable training, DMD requires an additional
regression loss computed using a large set of noise-image pairs generated by
the teacher with many steps of a deterministic sampler. This is costly for
large-scale text-to-image synthesis and limits the student's quality, tying it
too closely to the teacher's original sampling paths. We introduce DMD2, a set
of techniques that lift this limitation and improve DMD training. First, we
eliminate the regression loss and the need for expensive dataset construction.
We show that the resulting instability is due to the fake critic not estimating
the distribution of generated samples accurately and propose a two time-scale
update rule as a remedy. Second, we integrate a GAN loss into the distillation
procedure, discriminating between generated samples and real images. This lets
us train the student model on real data, mitigating the imperfect real score
estimation from the teacher model, and enhancing quality. Lastly, we modify the
training procedure to enable multi-step sampling. We identify and address the
training-inference input mismatch problem in this setting, by simulating
inference-time generator samples during training time. Taken together, our
improvements set new benchmarks in one-step image generation, with FID scores
of 1.28 on ImageNet-64x64 and 8.35 on zero-shot COCO 2014, surpassing the
original teacher despite a 500X reduction in inference cost. Further, we show
our approach can generate megapixel images by distilling SDXL, demonstrating
exceptional visual quality among few-step methods.
DMD2 enhances one-step image generation by removing regression loss, integrating GAN loss, and enabling multi-step sampling, setting new benchmarks and achieving exceptional visual quality.
diffusion models Distribution Matching Distillation (DMD) one-step generators noise-image pairs deterministic sampler two time-scale update rule GAN loss multi-step sampling training-inference input mismatch FID scores ImageNet-64x64 COCO 2014 megapixel images SDXL