Repulsive Score Distillation for Diverse Sampling of Diffusion Models
Score distillation sampling has been pivotal for integrating diffusion models
into generation of complex visuals. Despite impressive results it suffers from
mode collapse and lack of diversity. To cope with this challenge, we leverage
the gradient flow interpretation of score distillation to propose Repulsive
Score Distillation (RSD). In particular, we propose a variational framework
based on repulsion of an ensemble of particles that promotes diversity. Using a
variational approximation that incorporates a coupling among particles, the
repulsion appears as a simple regularization that allows interaction of
particles based on their relative pairwise similarity, measured e.g., via
radial basis kernels. We design RSD for both unconstrained and constrained
sampling scenarios. For constrained sampling we focus on inverse problems in
the latent space that leads to an augmented variational formulation, that
strikes a good balance between compute, quality and diversity. Our extensive
experiments for text-to-image generation, and inverse problems demonstrate that
RSD achieves a superior trade-off between diversity and quality compared with
state-of-the-art alternatives.
667b3187d24677e9af84110a
Nicolas Zilberstein
extracted_confirmed
2024-06-25T22:53:12.740Z
6635bc694b8c6ce85cc10c75
/avatars/ddbdfa1940d2858065d62b7ac597b48a.svg
Zilberstein
user
nicozilber
667b3187d24677e9af84110b
Morteza Mardani
667b3187d24677e9af84110c
Santiago Segarra
Repulsive Score Distillation for Diverse Sampling of Diffusion Models
Score distillation sampling has been pivotal for integrating diffusion models
into generation of complex visuals. Despite impressive results it suffers from
mode collapse and lack of diversity. To cope with this challenge, we leverage
the gradient flow interpretation of score distillation to propose Repulsive
Score Distillation (RSD). In particular, we propose a variational framework
based on repulsion of an ensemble of particles that promotes diversity. Using a
variational approximation that incorporates a coupling among particles, the
repulsion appears as a simple regularization that allows interaction of
particles based on their relative pairwise similarity, measured e.g., via
radial basis kernels. We design RSD for both unconstrained and constrained
sampling scenarios. For constrained sampling we focus on inverse problems in
the latent space that leads to an augmented variational formulation, that
strikes a good balance between compute, quality and diversity. Our extensive
experiments for text-to-image generation, and inverse problems demonstrate that
RSD achieves a superior trade-off between diversity and quality compared with
state-of-the-art alternatives.
Repulsive Score Distillation (RSD) enhances diversity in diffusion model sampling by introducing particle repulsion, improving quality in text-to-image generation and inverse problems.
score distillation mode collapse diffusion models gradient flow variational framework repulsion particles radial basis kernels unconstrained sampling constrained sampling inverse problems latent space augmented variational formulation diversity quality