Immiscible Diffusion: Accelerating Diffusion Training with Noise
  Assignment
In this paper, we point out suboptimal noise-data mapping leads to slow
training of diffusion models. During diffusion training, current methods
diffuse each image across the entire noise space, resulting in a mixture of all
images at every point in the noise layer. We emphasize that this random mixture
of noise-data mapping complicates the optimization of the denoising function in
diffusion models. Drawing inspiration from the immiscible phenomenon in
physics, we propose Immiscible Diffusion, a simple and effective method to
improve the random mixture of noise-data mapping. In physics, miscibility can
vary according to various intermolecular forces. Thus, immiscibility means that
the mixing of the molecular sources is distinguishable. Inspired by this, we
propose an assignment-then-diffusion training strategy. Specifically, prior to
diffusing the image data into noise, we assign diffusion target noise for the
image data by minimizing the total image-noise pair distance in a mini-batch.
The assignment functions analogously to external forces to separate the
diffuse-able areas of images, thus mitigating the inherent difficulties in
diffusion training. Our approach is remarkably simple, requiring only one line
of code to restrict the diffuse-able area for each image while preserving the
Gaussian distribution of noise. This ensures that each image is projected only
to nearby noise. To address the high complexity of the assignment algorithm, we
employ a quantized-assignment method to reduce the computational overhead to a
negligible level. Experiments demonstrate that our method achieve up to 3x
faster training for consistency models and DDIM on the CIFAR dataset, and up to
1.3x faster on CelebA datasets for consistency models. Besides, we conduct
thorough analysis about the Immiscible Diffusion, which sheds lights on how it
improves diffusion training speed while improving the fidelity.
66736978fad1f4ec2b3cb31d
Yiheng Li
claimed_verified
2025-05-30T07:00:34.345Z
65de6eacbeffeb39ba45a189
/avatars/fad81b9b9fadce96ba11d44d50366fb7.svg
Yiheng Li
user
yiheng-li
66736978fad1f4ec2b3cb31e
Heyang Jiang
admin_assigned
2024-06-20T09:47:11.977Z
65e0905b05caf76026a622f9
/avatars/d0ad6c94edd499645087e09583c8e41e.svg
Heyang Jiang
user
Firesheep
66736978fad1f4ec2b3cb31f
Akio Kodaira
claimed_verified
2025-07-08T08:09:03.882Z
63fedca388b9695964c33ad8
https://cdn-avatars.huggingface.co/v1/production/uploads/63fedca388b9695964c33ad8/XFE8Tm9yRRbo6XdDw7c2O.png
Aki
user
AkiCumulo
66736978fad1f4ec2b3cb320
Masayoshi Tomizuka
66736978fad1f4ec2b3cb321
Kurt Keutzer
admin_assigned
2024-06-20T09:46:59.171Z
6251bf4b183aa4266924ad91
https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
Kurt Keutzer
user
kurtkeutzer
66736978fad1f4ec2b3cb322
Chenfeng Xu
admin_assigned
2024-06-20T09:46:47.304Z
654ca71d5255ee86711b52c5
/avatars/52bf00fd74c8db5643c4daa185c678e6.svg
Chenfeng Xu
user
chenfengx
Immiscible Diffusion: Accelerating Diffusion Training with Noise
  Assignment
In this paper, we point out suboptimal noise-data mapping leads to slow
training of diffusion models. During diffusion training, current methods
diffuse each image across the entire noise space, resulting in a mixture of all
images at every point in the noise layer. We emphasize that this random mixture
of noise-data mapping complicates the optimization of the denoising function in
diffusion models. Drawing inspiration from the immiscible phenomenon in
physics, we propose Immiscible Diffusion, a simple and effective method to
improve the random mixture of noise-data mapping. In physics, miscibility can
vary according to various intermolecular forces. Thus, immiscibility means that
the mixing of the molecular sources is distinguishable. Inspired by this, we
propose an assignment-then-diffusion training strategy. Specifically, prior to
diffusing the image data into noise, we assign diffusion target noise for the
image data by minimizing the total image-noise pair distance in a mini-batch.
The assignment functions analogously to external forces to separate the
diffuse-able areas of images, thus mitigating the inherent difficulties in
diffusion training. Our approach is remarkably simple, requiring only one line
of code to restrict the diffuse-able area for each image while preserving the
Gaussian distribution of noise. This ensures that each image is projected only
to nearby noise. To address the high complexity of the assignment algorithm, we
employ a quantized-assignment method to reduce the computational overhead to a
negligible level. Experiments demonstrate that our method achieve up to 3x
faster training for consistency models and DDIM on the CIFAR dataset, and up to
1.3x faster on CelebA datasets for consistency models. Besides, we conduct
thorough analysis about the Immiscible Diffusion, which sheds lights on how it
improves diffusion training speed while improving the fidelity.
Immiscible Diffusion, which assigns diffusion target noise to images based on minimizing image-noise pair distances, improves the training speed and fidelity of diffusion models.
noise-data mapping diffusion training denoising function immiscible assignment-then-diffusion diffusion target noise total image-noise pair distance mini-batch Gaussian distribution consistency models DDIM CIFAR dataset CelebA datasets diffusion training speed fidelity