Variance-Aware Likelihood Bounds for Diffusion Models with Arbitrary Noise
Yirong Shen, Lu GAN, Cong Ling
Likelihood-based generative models have found wide applications such as compression, semi-supervised learning, and other real-world scenarios. At the heart of many high-performing likelihood models lie perturbation-based approaches. However, likelihood estimation in such models often suffers from slow convergence and limited theoretical insight. In this paper, we derive a novel likelihood bound for noise-driven models to improve both the accuracy and efficiency of maximum likelihood estimation. Central to our approach is an extension of the Kullback–Leibler (KL) divergence–Fisher information connection to arbitrary noise perturbations, elevating it from a theoretical concept to a practical design principle. This enables researchers to select noise distributions that naturally reflect sensor artifacts, quantization effects, or heavy-tailed data, without sacrificing standard diffusion training methodologies. By treating diffusion mapping as a Gaussian channel, we can formulate an exact expression for the mismatched entropy between the data and the model. Remarkably, our bound achieves on-par likelihood estimation on CIFAR-10 (2.50 bits/dim) and state-of-the-art performance on ImageNet-32 (3.01 bits/dim), all without using any data augmentation. Furthermore, our theoretical formulation may also offer benefits for modelling discrete data distributions, suggesting broader applicability beyond continuous domains.