Large-scale Reinforcement Learning for Diffusion Models
Text-to-image diffusion models are a class of deep generative models that
have demonstrated an impressive capacity for high-quality image generation.
However, these models are susceptible to implicit biases that arise from
web-scale text-image training pairs and may inaccurately model aspects of
images we care about. This can result in suboptimal samples, model bias, and
images that do not align with human ethics and preferences. In this paper, we
present an effective scalable algorithm to improve diffusion models using
Reinforcement Learning (RL) across a diverse set of reward functions, such as
human preference, compositionality, and fairness over millions of images. We
illustrate how our approach substantially outperforms existing methods for
aligning diffusion models with human preferences. We further illustrate how
this substantially improves pretrained Stable Diffusion (SD) models, generating
samples that are preferred by humans 80.3% of the time over those from the base
SD model while simultaneously improving both the composition and diversity of
generated samples.
65b071e07febbcc2afc9d680
Yinan Zhang
65b071e07febbcc2afc9d681
Eric Tzeng
65b071e07febbcc2afc9d682
Yilun Du
admin_assigned
2024-01-24T09:15:33.839Z
63c9bd445fdc575773c732fe
/avatars/def472d1ab3fbf751225357c0932ae7e.svg
Yilun Du
user
yilundu
65b071e07febbcc2afc9d683
Dmitry Kislyuk
admin_assigned
2024-01-24T09:15:40.490Z
6383ce4411cd91c15d5c9a11
/avatars/aba18939f4744185f0b5fb9a1ba2ce03.svg
Dmitry Kislyuk
user
molkie
Large-scale Reinforcement Learning for Diffusion Models
Text-to-image diffusion models are a class of deep generative models that
have demonstrated an impressive capacity for high-quality image generation.
However, these models are susceptible to implicit biases that arise from
web-scale text-image training pairs and may inaccurately model aspects of
images we care about. This can result in suboptimal samples, model bias, and
images that do not align with human ethics and preferences. In this paper, we
present an effective scalable algorithm to improve diffusion models using
Reinforcement Learning (RL) across a diverse set of reward functions, such as
human preference, compositionality, and fairness over millions of images. We
illustrate how our approach substantially outperforms existing methods for
aligning diffusion models with human preferences. We further illustrate how
this substantially improves pretrained Stable Diffusion (SD) models, generating
samples that are preferred by humans 80.3% of the time over those from the base
SD model while simultaneously improving both the composition and diversity of
generated samples.
An algorithm using Reinforcement Learning improves text-to-image diffusion models by aligning with human preferences, enhancing sample quality, compositionality, and diversity.
diffusion models Reinforcement Learning reward functions human preference compositionality fairness Stable Diffusion sample quality sample diversity