Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models
Recently, diffusion models have made remarkable progress in text-to-image
(T2I) generation, synthesizing images with high fidelity and diverse contents.
Despite this advancement, latent space smoothness within diffusion models
remains largely unexplored. Smooth latent spaces ensure that a perturbation on
an input latent corresponds to a steady change in the output image. This
property proves beneficial in downstream tasks, including image interpolation,
inversion, and editing. In this work, we expose the non-smoothness of diffusion
latent spaces by observing noticeable visual fluctuations resulting from minor
latent variations. To tackle this issue, we propose Smooth Diffusion, a new
category of diffusion models that can be simultaneously high-performing and
smooth. Specifically, we introduce Step-wise Variation Regularization to
enforce the proportion between the variations of an arbitrary input latent and
that of the output image is a constant at any diffusion training step. In
addition, we devise an interpolation standard deviation (ISTD) metric to
effectively assess the latent space smoothness of a diffusion model. Extensive
quantitative and qualitative experiments demonstrate that Smooth Diffusion
stands out as a more desirable solution not only in T2I generation but also
across various downstream tasks. Smooth Diffusion is implemented as a
plug-and-play Smooth-LoRA to work with various community models. Code is
available at https://github.com/SHI-Labs/Smooth-Diffusion.
6572949959773fd21b605fc2
Jiayi Guo
admin_assigned
2023-12-08T10:33:01.016Z
6417c111dce1e4c0229ec182
/avatars/de65da1f02b136b1af57862d9d0c48f2.svg
Jiayi Guo
user
JiayiGuo821
6572949959773fd21b605fc3
Xingqian Xu
admin_assigned
2023-12-08T10:33:07.349Z
6341a3ebc5565a4b8d45db7f
https://cdn-avatars.huggingface.co/v1/production/uploads/1668583035141-6341a3ebc5565a4b8d45db7f.jpeg
Xingqian Xu
user
JamesXu
6572949959773fd21b605fc4
Yifan Pu
admin_assigned
2023-12-08T10:33:13.836Z
647e83257f9ad5e44babe82a
/avatars/2d9593775c49856fe5dfa5bd23dfcda7.svg
yifan pu
user
yifanpu001
6572949959773fd21b605fc5
Zanlin Ni
admin_assigned
2023-12-08T10:33:19.983Z
63987ffb2ceb55aabe0852f3
/avatars/343b796ff6b8906203904e8c620d7eb5.svg
Zanlin Ni
user
nzl-thu
6572949959773fd21b605fc6
Chaofei Wang
admin_assigned
2023-12-08T10:33:26.667Z
651c3def7d3cfa8bcd768339
/avatars/a45cb0f7acf35cf40fe18e6dfc1db112.svg
Chaofei Wang
user
wangcf
6572949959773fd21b605fc7
Manushree Vasu
6572949959773fd21b605fc8
Shiji Song
admin_assigned
2023-12-08T10:33:46.576Z
6497a53dbcde24ff442391b0
/avatars/2e4957f8426223c9233f45414737b7c0.svg
宋世杰
user
songshijie
6572949959773fd21b605fc9
Gao Huang
6572949959773fd21b605fca
Humphrey Shi
admin_assigned
2023-12-08T10:34:07.019Z
61e1188afc27c0f5e3641eb3
https://cdn-avatars.huggingface.co/v1/production/uploads/61e1188afc27c0f5e3641eb3/R4JFlWgYomZFvZ9CLSZfD.jpeg
Humphrey Shi
user
Humphrey
Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models
Recently, diffusion models have made remarkable progress in text-to-image
(T2I) generation, synthesizing images with high fidelity and diverse contents.
Despite this advancement, latent space smoothness within diffusion models
remains largely unexplored. Smooth latent spaces ensure that a perturbation on
an input latent corresponds to a steady change in the output image. This
property proves beneficial in downstream tasks, including image interpolation,
inversion, and editing. In this work, we expose the non-smoothness of diffusion
latent spaces by observing noticeable visual fluctuations resulting from minor
latent variations. To tackle this issue, we propose Smooth Diffusion, a new
category of diffusion models that can be simultaneously high-performing and
smooth. Specifically, we introduce Step-wise Variation Regularization to
enforce the proportion between the variations of an arbitrary input latent and
that of the output image is a constant at any diffusion training step. In
addition, we devise an interpolation standard deviation (ISTD) metric to
effectively assess the latent space smoothness of a diffusion model. Extensive
quantitative and qualitative experiments demonstrate that Smooth Diffusion
stands out as a more desirable solution not only in T2I generation but also
across various downstream tasks. Smooth Diffusion is implemented as a
plug-and-play Smooth-LoRA to work with various community models. Code is
available at https://github.com/SHI-Labs/Smooth-Diffusion.
Smooth Diffusion addresses non-smooth latent spaces in diffusion models by enforcing consistent variation ratios, improving tasks like text-to-image generation, interpolation, inversion, and editing.
diffusion models text-to-image generation latent space smoothness Step-wise Variation Regularization interpolation standard deviation (ISTD) Smooth-LoRA