The Superposition of Diffusion Models Using the Itô Density Estimator
The Cambrian explosion of easily accessible pre-trained diffusion models
suggests a demand for methods that combine multiple different pre-trained
diffusion models without incurring the significant computational burden of
re-training a larger combined model. In this paper, we cast the problem of
combining multiple pre-trained diffusion models at the generation stage under a
novel proposed framework termed superposition. Theoretically, we derive
superposition from rigorous first principles stemming from the celebrated
continuity equation and design two novel algorithms tailor-made for combining
diffusion models in SuperDiff. SuperDiff leverages a new scalable It\^o density
estimator for the log likelihood of the diffusion SDE which incurs no
additional overhead compared to the well-known Hutchinson's estimator needed
for divergence calculations. We demonstrate that SuperDiff is scalable to large
pre-trained diffusion models as superposition is performed solely through
composition during inference, and also enjoys painless implementation as it
combines different pre-trained vector fields through an automated re-weighting
scheme. Notably, we show that SuperDiff is efficient during inference time, and
mimics traditional composition operators such as the logical OR and the logical
AND. We empirically demonstrate the utility of using SuperDiff for generating
more diverse images on CIFAR-10, more faithful prompt conditioned image editing
using Stable Diffusion, and improved unconditional de novo structure design of
proteins. https://github.com/necludov/super-diffusion
676d0a2e0076ad5ba195b88a
Marta Skreta
claimed_verified
2024-12-30T19:32:04.997Z
65a49ef27ec6af0f956a5c61
https://cdn-avatars.huggingface.co/v1/production/uploads/65a49ef27ec6af0f956a5c61/OGJ1OVezooF7dUn1SiEKG.jpeg
Marta Skreta
user
mskrt
676d0a2e0076ad5ba195b88b
Lazar Atanackovic
claimed_verified
2024-12-30T19:32:02.707Z
66cd0e523f6dbc62985b184b
https://cdn-avatars.huggingface.co/v1/production/uploads/66cd0e523f6dbc62985b184b/vlHTY9LKJ0jKeAniHdGil.jpeg
Lazar Atanackovic
user
lazaratan
676d0a2e0076ad5ba195b88c
Avishek Joey Bose
676d0a2e0076ad5ba195b88d
Alexander Tong
676d0a2e0076ad5ba195b88e
Kirill Neklyudov
claimed_verified
2024-12-30T20:44:53.406Z
65039d7a93574a897179139a
https://cdn-avatars.huggingface.co/v1/production/uploads/65039d7a93574a897179139a/szQAWkF2M9XnYKykmX8Cg.jpeg
Kirill Neklyudov
user
necludov
The Superposition of Diffusion Models Using the Itô Density Estimator
The Cambrian explosion of easily accessible pre-trained diffusion models
suggests a demand for methods that combine multiple different pre-trained
diffusion models without incurring the significant computational burden of
re-training a larger combined model. In this paper, we cast the problem of
combining multiple pre-trained diffusion models at the generation stage under a
novel proposed framework termed superposition. Theoretically, we derive
superposition from rigorous first principles stemming from the celebrated
continuity equation and design two novel algorithms tailor-made for combining
diffusion models in SuperDiff. SuperDiff leverages a new scalable It\^o density
estimator for the log likelihood of the diffusion SDE which incurs no
additional overhead compared to the well-known Hutchinson's estimator needed
for divergence calculations. We demonstrate that SuperDiff is scalable to large
pre-trained diffusion models as superposition is performed solely through
composition during inference, and also enjoys painless implementation as it
combines different pre-trained vector fields through an automated re-weighting
scheme. Notably, we show that SuperDiff is efficient during inference time, and
mimics traditional composition operators such as the logical OR and the logical
AND. We empirically demonstrate the utility of using SuperDiff for generating
more diverse images on CIFAR-10, more faithful prompt conditioned image editing
using Stable Diffusion, and improved unconditional de novo structure design of
proteins. https://github.com/necludov/super-diffusion
SuperDiff framework combines multiple pre-trained diffusion models efficiently during inference without re-training, enabling diverse image generation and protein structure design.
diffusion models superposition continuity equation It^o density estimator divergence calculations SuperDiff inference re-weighting scheme logical OR logical AND CIFAR-10 Stable Diffusion protein structure design