Margin-aware Preference Optimization for Aligning Diffusion Models
  without Reference
Modern alignment techniques based on human preferences, such as RLHF and DPO,
typically employ divergence regularization relative to the reference model to
ensure training stability. However, this often limits the flexibility of models
during alignment, especially when there is a clear distributional discrepancy
between the preference data and the reference model. In this paper, we focus on
the alignment of recent text-to-image diffusion models, such as Stable
Diffusion XL (SDXL), and find that this "reference mismatch" is indeed a
significant problem in aligning these models due to the unstructured nature of
visual modalities: e.g., a preference for a particular stylistic aspect can
easily induce such a discrepancy. Motivated by this observation, we propose a
novel and memory-friendly preference alignment method for diffusion models that
does not depend on any reference model, coined margin-aware preference
optimization (MaPO). MaPO jointly maximizes the likelihood margin between the
preferred and dispreferred image sets and the likelihood of the preferred sets,
simultaneously learning general stylistic features and preferences. For
evaluation, we introduce two new pairwise preference datasets, which comprise
self-generated image pairs from SDXL, Pick-Style and Pick-Safety, simulating
diverse scenarios of reference mismatch. Our experiments validate that MaPO can
significantly improve alignment on Pick-Style and Pick-Safety and general
preference alignment when used with Pick-a-Pic v2, surpassing the base SDXL and
other existing methods. Our code, models, and datasets are publicly available
via https://mapo-t2i.github.io
6667e515fd95ddf66cab8fe8
Jiwoo Hong
claimed_verified
2024-06-11T13:26:34.509Z
6415c043486c7c9a5d151583
https://cdn-avatars.huggingface.co/v1/production/uploads/6415c043486c7c9a5d151583/fUdYFh6iVh57swCkBEy-y.jpeg
Jiwoo Hong
user
JW17
6667e515fd95ddf66cab8fe9
Sayak Paul
claimed_verified
2024-06-11T09:21:16.755Z
5f7fbd813e94f16a85448745
https://cdn-avatars.huggingface.co/v1/production/uploads/1649681653581-5f7fbd813e94f16a85448745.jpeg
Sayak Paul
user
sayakpaul
6667e515fd95ddf66cab8fea
Noah Lee
claimed_verified
2024-09-20T07:48:42.960Z
6296fe6f7d586f66c3957f33
https://cdn-avatars.huggingface.co/v1/production/uploads/6296fe6f7d586f66c3957f33/lCHe1mBVRzQ1sKsXD2OUd.jpeg
Noah Lee
user
nlee-208
6667e515fd95ddf66cab8feb
Kashif Rasul
admin_assigned
2024-06-11T15:13:41.779Z
629f3b18ee05727ce328ccbe
https://cdn-avatars.huggingface.co/v1/production/uploads/1669189789447-629f3b18ee05727ce328ccbe.jpeg
Kashif Rasul
user
kashif
6667e515fd95ddf66cab8fec
James Thorne
6667e515fd95ddf66cab8fed
Jongheon Jeong
claimed_verified
2024-06-12T07:30:12.340Z
660f22b4a8c77694a9c49486
/avatars/718727912a7fdebb7daf696776746839.svg
Jongheon Jeong
user
jh-jeong
Margin-aware Preference Optimization for Aligning Diffusion Models
  without Reference
Modern alignment techniques based on human preferences, such as RLHF and DPO,
typically employ divergence regularization relative to the reference model to
ensure training stability. However, this often limits the flexibility of models
during alignment, especially when there is a clear distributional discrepancy
between the preference data and the reference model. In this paper, we focus on
the alignment of recent text-to-image diffusion models, such as Stable
Diffusion XL (SDXL), and find that this "reference mismatch" is indeed a
significant problem in aligning these models due to the unstructured nature of
visual modalities: e.g., a preference for a particular stylistic aspect can
easily induce such a discrepancy. Motivated by this observation, we propose a
novel and memory-friendly preference alignment method for diffusion models that
does not depend on any reference model, coined margin-aware preference
optimization (MaPO). MaPO jointly maximizes the likelihood margin between the
preferred and dispreferred image sets and the likelihood of the preferred sets,
simultaneously learning general stylistic features and preferences. For
evaluation, we introduce two new pairwise preference datasets, which comprise
self-generated image pairs from SDXL, Pick-Style and Pick-Safety, simulating
diverse scenarios of reference mismatch. Our experiments validate that MaPO can
significantly improve alignment on Pick-Style and Pick-Safety and general
preference alignment when used with Pick-a-Pic v2, surpassing the base SDXL and
other existing methods. Our code, models, and datasets are publicly available
via https://mapo-t2i.github.io
A novel memory-friendly preference alignment method, MaPO, improves the alignment of text-to-image diffusion models like Stable Diffusion XL by maximizing the likelihood margin between preferred and dispreferred images without relying on a reference model.
RLHF DPO divergence regularization reference model alignment text-to-image diffusion models Stable Diffusion XL reference mismatch visual modalities margin-aware preference optimization likelihood margin pairwise preference datasets Pick-Style Pick-Safety Pick-a-Pic v2