Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human
  Preference
Recent studies have demonstrated the effectiveness of directly aligning
diffusion models with human preferences using differentiable reward. However,
they exhibit two primary challenges: (1) they rely on multistep denoising with
gradient computation for reward scoring, which is computationally expensive,
thus restricting optimization to only a few diffusion steps; (2) they often
need continuous offline adaptation of reward models in order to achieve desired
aesthetic quality, such as photorealism or precise lighting effects. To address
the limitation of multistep denoising, we propose Direct-Align, a method that
predefines a noise prior to effectively recover original images from any time
steps via interpolation, leveraging the equation that diffusion states are
interpolations between noise and target images, which effectively avoids
over-optimization in late timesteps. Furthermore, we introduce Semantic
Relative Preference Optimization (SRPO), in which rewards are formulated as
text-conditioned signals. This approach enables online adjustment of rewards in
response to positive and negative prompt augmentation, thereby reducing the
reliance on offline reward fine-tuning. By fine-tuning the FLUX.1.dev model
with optimized denoising and online reward adjustment, we improve its
human-evaluated realism and aesthetic quality by over 3x.
68bfd5e9207285de11b07d9a
Xiangwei Shen
68bfd5e9207285de11b07d9b
Zhimin Li
68bfd5e9207285de11b07d9c
Zhantao Yang
68bfd5e9207285de11b07d9d
Shiyi Zhang
68bfd5e9207285de11b07d9e
Yingfang Zhang
68bfd5e9207285de11b07d9f
Donghao Li
68bfd5e9207285de11b07da0
Chunyu Wang
68bfd5e9207285de11b07da1
Qinglin Lu
68bfd5e9207285de11b07da2
Yansong Tang
Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human
  Preference
Recent studies have demonstrated the effectiveness of directly aligning
diffusion models with human preferences using differentiable reward. However,
they exhibit two primary challenges: (1) they rely on multistep denoising with
gradient computation for reward scoring, which is computationally expensive,
thus restricting optimization to only a few diffusion steps; (2) they often
need continuous offline adaptation of reward models in order to achieve desired
aesthetic quality, such as photorealism or precise lighting effects. To address
the limitation of multistep denoising, we propose Direct-Align, a method that
predefines a noise prior to effectively recover original images from any time
steps via interpolation, leveraging the equation that diffusion states are
interpolations between noise and target images, which effectively avoids
over-optimization in late timesteps. Furthermore, we introduce Semantic
Relative Preference Optimization (SRPO), in which rewards are formulated as
text-conditioned signals. This approach enables online adjustment of rewards in
response to positive and negative prompt augmentation, thereby reducing the
reliance on offline reward fine-tuning. By fine-tuning the FLUX.1.dev model
with optimized denoising and online reward adjustment, we improve its
human-evaluated realism and aesthetic quality by over 3x.
Direct-Align and Semantic Relative Preference Optimization improve diffusion models' alignment with human preferences by reducing computational costs and minimizing offline reward adaptation.
diffusion models differentiable reward multistep denoising gradient computation noise prior interpolation diffusion states Semantic Relative Preference Optimization text-conditioned signals prompt augmentation FLUX.1.dev model human-evaluated realism aesthetic quality