Locality in Image Diffusion Models Emerges from Data Statistics
Among generative models, diffusion models are uniquely intriguing due to the
existence of a closed-form optimal minimizer of their training objective, often
referred to as the optimal denoiser. However, diffusion using this optimal
denoiser merely reproduces images in the training set and hence fails to
capture the behavior of deep diffusion models. Recent work has attempted to
characterize this gap between the optimal denoiser and deep diffusion models,
proposing analytical, training-free models that can generate images that
resemble those generated by a trained UNet. The best-performing method
hypothesizes that shift equivariance and locality inductive biases of
convolutional neural networks are the cause of the performance gap, hence
incorporating these assumptions into its analytical model. In this work, we
present evidence that the locality in deep diffusion models emerges as a
statistical property of the image dataset, not due to the inductive bias of
convolutional neural networks. Specifically, we demonstrate that an optimal
parametric linear denoiser exhibits similar locality properties to the deep
neural denoisers. We further show, both theoretically and experimentally, that
this locality arises directly from the pixel correlations present in natural
image datasets. Finally, we use these insights to craft an analytical denoiser
that better matches scores predicted by a deep diffusion model than the prior
expert-crafted alternative.
68c8d0fd733e345e52ac1ea5
Artem Lukoianov
68c8d0fd733e345e52ac1ea6
Chenyang Yuan
68c8d0fd733e345e52ac1ea7
Justin Solomon
68c8d0fd733e345e52ac1ea8
Vincent Sitzmann
Locality in Image Diffusion Models Emerges from Data Statistics
Among generative models, diffusion models are uniquely intriguing due to the
existence of a closed-form optimal minimizer of their training objective, often
referred to as the optimal denoiser. However, diffusion using this optimal
denoiser merely reproduces images in the training set and hence fails to
capture the behavior of deep diffusion models. Recent work has attempted to
characterize this gap between the optimal denoiser and deep diffusion models,
proposing analytical, training-free models that can generate images that
resemble those generated by a trained UNet. The best-performing method
hypothesizes that shift equivariance and locality inductive biases of
convolutional neural networks are the cause of the performance gap, hence
incorporating these assumptions into its analytical model. In this work, we
present evidence that the locality in deep diffusion models emerges as a
statistical property of the image dataset, not due to the inductive bias of
convolutional neural networks. Specifically, we demonstrate that an optimal
parametric linear denoiser exhibits similar locality properties to the deep
neural denoisers. We further show, both theoretically and experimentally, that
this locality arises directly from the pixel correlations present in natural
image datasets. Finally, we use these insights to craft an analytical denoiser
that better matches scores predicted by a deep diffusion model than the prior
expert-crafted alternative.
Research shows that locality in deep diffusion models is a statistical property of image datasets rather than an inductive bias of convolutional neural networks, leading to the development of a more accurate analytical denoiser.
diffusion models optimal denoiser UNet shift equivariance locality inductive biases convolutional neural networks parametric linear denoiser pixel correlations natural image datasets analytical denoiser