Analyzing and Improving the Training Dynamics of Diffusion Models
Diffusion models currently dominate the field of data-driven image synthesis
with their unparalleled scaling to large datasets. In this paper, we identify
and rectify several causes for uneven and ineffective training in the popular
ADM diffusion model architecture, without altering its high-level structure.
Observing uncontrolled magnitude changes and imbalances in both the network
activations and weights over the course of training, we redesign the network
layers to preserve activation, weight, and update magnitudes on expectation. We
find that systematic application of this philosophy eliminates the observed
drifts and imbalances, resulting in considerably better networks at equal
computational complexity. Our modifications improve the previous record FID of
2.41 in ImageNet-512 synthesis to 1.81, achieved using fast deterministic
sampling.
  As an independent contribution, we present a method for setting the
exponential moving average (EMA) parameters post-hoc, i.e., after completing
the training run. This allows precise tuning of EMA length without the cost of
performing several training runs, and reveals its surprising interactions with
network architecture, training time, and guidance.
656fe86f2f058b368c2b06e7
Tero Karras
extracted_confirmed
2024-05-30T13:22:25.068Z
66587947b7d313010028ac02
/avatars/cb95f524dd1c80aa6ec8af977c5f2606.svg
Tero Karras
user
tkarras
656fe86f2f058b368c2b06e8
Miika Aittala
admin_assigned
2023-12-06T11:54:46.315Z
630f80e2a119d49bc1e3836f
/avatars/e0ffe1c19df64606cb30801ad20a9f43.svg
Miika Aittala
user
miika
656fe86f2f058b368c2b06e9
Jaakko Lehtinen
656fe86f2f058b368c2b06ea
Janne Hellsten
admin_assigned
2023-12-06T11:54:55.806Z
63dc13a562a61013a7509c17
/avatars/b537cf5459a452c4e61b2a8f404c18db.svg
Janne Hellsten
user
Jlou
656fe86f2f058b368c2b06eb
Timo Aila
656fe86f2f058b368c2b06ec
Samuli Laine
extracted_confirmed
2025-02-12T11:41:19.367Z
67ac68cda78e0ee095322704
/avatars/7cc9fbd7ba8c02e48b01e56bbccea01e.svg
Samuli Laine
user
slaine2
Analyzing and Improving the Training Dynamics of Diffusion Models
Diffusion models currently dominate the field of data-driven image synthesis
with their unparalleled scaling to large datasets. In this paper, we identify
and rectify several causes for uneven and ineffective training in the popular
ADM diffusion model architecture, without altering its high-level structure.
Observing uncontrolled magnitude changes and imbalances in both the network
activations and weights over the course of training, we redesign the network
layers to preserve activation, weight, and update magnitudes on expectation. We
find that systematic application of this philosophy eliminates the observed
drifts and imbalances, resulting in considerably better networks at equal
computational complexity. Our modifications improve the previous record FID of
2.41 in ImageNet-512 synthesis to 1.81, achieved using fast deterministic
sampling.
  As an independent contribution, we present a method for setting the
exponential moving average (EMA) parameters post-hoc, i.e., after completing
the training run. This allows precise tuning of EMA length without the cost of
performing several training runs, and reveals its surprising interactions with
network architecture, training time, and guidance.
Modifications to network layers in the ADM diffusion model architecture improve training stability and synthesis quality, reducing FID from 2.41 to 1.81, and a novel method for post-hoc EMA parameter tuning is introduced.
diffusion models ADM diffusion model network activations weights fast deterministic sampling exponential moving average (EMA)