Information-Theoretic Discrete Diffusion
Moongyu Jeon, Sangwoo Shin, Dongjae Jeon, Albert No
We present an information-theoretic framework for discrete diffusion models, introducing principled estimators of log-likelihood based on score-matching losses.Inspired by the I-MMSE identity for Gaussian denoising models, we derive analogous results for discrete diffusion.Specifically, we establish the Information–Minimum Denoising Score Entropy (I-MDSE) relation,which links the mutual information between data and its diffused counterpart to the minimum denoising score entropy (DSE) loss.We extend this theory to the masked diffusion setting and introduce the Information–Minimum Denoising Cross-Entropy (I-MDCE) relation,which connects cross-entropy-based losses to information flow in absorbing diffusion processes.These results yield closed-form decompositions of the negative log-likelihood in both time-integral and time-free forms,enabling efficient estimation of data likelihood.We further show that these decompositions naturally extend to conditional likelihoods, allowing direct estimation of $\log p_0(\mathbf{x}^\text{target}|\mathbf{x}^\text{context})$ in structured tasks such as prompt–response modeling.Empirical results on synthetic and real-world data confirm that the proposed approach accurately estimates log-probabilities of discrete inputs.