Aligning Diffusion Models with Noise-Conditioned Perception
Recent advancements in human preference optimization, initially developed for
Language Models (LMs), have shown promise for text-to-image Diffusion Models,
enhancing prompt alignment, visual appeal, and user preference. Unlike LMs,
Diffusion Models typically optimize in pixel or VAE space, which does not align
well with human perception, leading to slower and less efficient training
during the preference alignment stage. We propose using a perceptual objective
in the U-Net embedding space of the diffusion model to address these issues.
Our approach involves fine-tuning Stable Diffusion 1.5 and XL using Direct
Preference Optimization (DPO), Contrastive Preference Optimization (CPO), and
supervised fine-tuning (SFT) within this embedding space. This method
significantly outperforms standard latent-space implementations across various
metrics, including quality and computational cost. For SDXL, our approach
provides 60.8\% general preference, 62.2\% visual appeal, and 52.1\% prompt
following against original open-sourced SDXL-DPO on the PartiPrompts dataset,
while significantly reducing compute. Our approach not only improves the
efficiency and quality of human preference alignment for diffusion models but
is also easily integrable with other optimization techniques. The training code
and LoRA weights will be available here:
https://huggingface.co/alexgambashidze/SDXL\_NCP-DPO\_v0.1
667bb9fcb9e88cc05404e766
Alexander Gambashidze
extracted_confirmed
2024-06-27T19:04:13.562Z
6192657ba9638054a9818f04
/avatars/35f6f6eb2e8f6b283034632a141c2670.svg
Alexander Gambashidze
user
alexgambashidze
667bb9fcb9e88cc05404e767
Anton Kulikov
admin_assigned
2024-06-26T10:00:33.181Z
6650c0bd87144bf8106682fb
/avatars/6f241c967b045c6d18fba8efd0fee3c6.svg
Anton Kulikov
user
neolizee
667bb9fcb9e88cc05404e768
Yuriy Sosnin
claimed_verified
2024-06-27T09:03:14.183Z
6576ea2e66330867a1fd3fc2
/avatars/27553edc0693ca6559437a05f3b5fb21.svg
Yuriy Sosnin
user
yuasosnin
667bb9fcb9e88cc05404e769
Ilya Makarov
Aligning Diffusion Models with Noise-Conditioned Perception
Recent advancements in human preference optimization, initially developed for
Language Models (LMs), have shown promise for text-to-image Diffusion Models,
enhancing prompt alignment, visual appeal, and user preference. Unlike LMs,
Diffusion Models typically optimize in pixel or VAE space, which does not align
well with human perception, leading to slower and less efficient training
during the preference alignment stage. We propose using a perceptual objective
in the U-Net embedding space of the diffusion model to address these issues.
Our approach involves fine-tuning Stable Diffusion 1.5 and XL using Direct
Preference Optimization (DPO), Contrastive Preference Optimization (CPO), and
supervised fine-tuning (SFT) within this embedding space. This method
significantly outperforms standard latent-space implementations across various
metrics, including quality and computational cost. For SDXL, our approach
provides 60.8\% general preference, 62.2\% visual appeal, and 52.1\% prompt
following against original open-sourced SDXL-DPO on the PartiPrompts dataset,
while significantly reducing compute. Our approach not only improves the
efficiency and quality of human preference alignment for diffusion models but
is also easily integrable with other optimization techniques. The training code
and LoRA weights will be available here:
https://huggingface.co/alexgambashidze/SDXL\_NCP-DPO\_v0.1
Using perceptual objectives in U-Net embedding space for text-to-image diffusion models enhances human preference alignment, visual appeal, and efficiency compared to traditional methods.
human preference optimization text-to-image diffusion models perceptual objective U-Net embedding space Stable Diffusion Direct Preference Optimization Contrastive Preference Optimization supervised fine-tuning general preference visual appeal prompt following PartiPrompts dataset computational cost LoRA weights