Dale meets Langevin: A Multiplicative Denoising Diffusion Model
Gradient descent has proven to be a powerful and effective technique for
optimization in numerous machine learning applications. Recent advances in
computational neuroscience have shown that learning in standard gradient
descent optimization formulation is not consistent with learning in biological
systems. This has opened up interesting avenues for building biologically
inspired learning techniques. One such approach is inspired by Dale's law,
which states that inhibitory and excitatory synapses do not swap roles during
the course of learning. The resulting exponential gradient descent optimization
scheme leads to log-normally distributed synaptic weights. Interestingly, the
density that satisfies the Fokker-Planck equation corresponding to the
stochastic differential equation (SDE) with geometric Brownian motion (GBM) is
the log-normal density. Leveraging this connection, we start with the SDE
governing geometric Brownian motion, and show that discretizing the
corresponding reverse-time SDE yields a multiplicative update rule, which
surprisingly, coincides with the sampling equivalent of the exponential
gradient descent update founded on Dale's law. Furthermore, we propose a new
formalism for multiplicative denoising score-matching, subsuming the loss
function proposed by Hyvaerinen for non-negative data. Indeed, log-normally
distributed data is positive and the proposed score-matching formalism turns
out to be a natural fit. This allows for training of score-based models for
image data and results in a novel multiplicative update scheme for sample
generation starting from a log-normal density. Experimental results on MNIST,
Fashion MNIST, and Kuzushiji datasets demonstrate generative capability of the
new scheme. To the best of our knowledge, this is the first instance of a
biologically inspired generative model employing multiplicative updates,
founded on geometric Brownian motion.
68e3764673e20ab577842108
Nishanth Shetty
claimed_verified
2025-10-07T12:28:23.589Z
637c6ce69495870ef76f7dbf
https://cdn-avatars.huggingface.co/v1/production/uploads/637c6ce69495870ef76f7dbf/48GgkvkdXI4CIT3gD76mh.jpeg
Nishanth Shetty
user
psitama
68e3764673e20ab577842109
Madhava Prasath
claimed_verified
2025-10-07T12:28:26.149Z
649c0ce8b4ee3b2664590a40
/avatars/090c1f57c29af918e100db0844f60892.svg
Madhava prasath
user
Madddy
68e3764673e20ab57784210a
Chandra Sekhar Seelamantula
Dale meets Langevin: A Multiplicative Denoising Diffusion Model
Gradient descent has proven to be a powerful and effective technique for
optimization in numerous machine learning applications. Recent advances in
computational neuroscience have shown that learning in standard gradient
descent optimization formulation is not consistent with learning in biological
systems. This has opened up interesting avenues for building biologically
inspired learning techniques. One such approach is inspired by Dale's law,
which states that inhibitory and excitatory synapses do not swap roles during
the course of learning. The resulting exponential gradient descent optimization
scheme leads to log-normally distributed synaptic weights. Interestingly, the
density that satisfies the Fokker-Planck equation corresponding to the
stochastic differential equation (SDE) with geometric Brownian motion (GBM) is
the log-normal density. Leveraging this connection, we start with the SDE
governing geometric Brownian motion, and show that discretizing the
corresponding reverse-time SDE yields a multiplicative update rule, which
surprisingly, coincides with the sampling equivalent of the exponential
gradient descent update founded on Dale's law. Furthermore, we propose a new
formalism for multiplicative denoising score-matching, subsuming the loss
function proposed by Hyvaerinen for non-negative data. Indeed, log-normally
distributed data is positive and the proposed score-matching formalism turns
out to be a natural fit. This allows for training of score-based models for
image data and results in a novel multiplicative update scheme for sample
generation starting from a log-normal density. Experimental results on MNIST,
Fashion MNIST, and Kuzushiji datasets demonstrate generative capability of the
new scheme. To the best of our knowledge, this is the first instance of a
biologically inspired generative model employing multiplicative updates,
founded on geometric Brownian motion.
A biologically inspired generative model using multiplicative updates based on geometric Brownian motion and exponential gradient descent achieves state-of-the-art performance on image datasets.
gradient descent exponential gradient descent Dale's law synaptic weights Fokker-Planck equation stochastic differential equation geometric Brownian motion multiplicative update rule denoising score-matching score-based models log-normal density