ID-Aligner: Enhancing Identity-Preserving Text-to-Image Generation with
  Reward Feedback Learning
The rapid development of diffusion models has triggered diverse applications.
Identity-preserving text-to-image generation (ID-T2I) particularly has received
significant attention due to its wide range of application scenarios like AI
portrait and advertising. While existing ID-T2I methods have demonstrated
impressive results, several key challenges remain: (1) It is hard to maintain
the identity characteristics of reference portraits accurately, (2) The
generated images lack aesthetic appeal especially while enforcing identity
retention, and (3) There is a limitation that cannot be compatible with
LoRA-based and Adapter-based methods simultaneously. To address these issues,
we present ID-Aligner, a general feedback learning framework to
enhance ID-T2I performance. To resolve identity features lost, we introduce
identity consistency reward fine-tuning to utilize the feedback from face
detection and recognition models to improve generated identity preservation.
Furthermore, we propose identity aesthetic reward fine-tuning leveraging
rewards from human-annotated preference data and automatically constructed
feedback on character structure generation to provide aesthetic tuning signals.
Thanks to its universal feedback fine-tuning framework, our method can be
readily applied to both LoRA and Adapter models, achieving consistent
performance gains. Extensive experiments on SD1.5 and SDXL diffusion models
validate the effectiveness of our approach. Project Page:
\url{https://idaligner.github.io/}
662a7078c9652518e153a734
Weifeng Chen
claimed_verified
2024-04-26T07:42:54.212Z
62a7fc03ac97233f161c2433
https://cdn-avatars.huggingface.co/v1/production/uploads/62a7fc03ac97233f161c2433/19wL1aCwyR-ik2WDXvns9.jpeg
weifeng
user
weifeng-chen
662a7078c9652518e153a735
Jiacheng Zhang
662a7078c9652518e153a736
Jie Wu
claimed_verified
2024-04-28T19:30:32.789Z
6381c5d63680a7cf34e08ca9
/avatars/731467e2d80d0ae163c4a00a9e3ff9e5.svg
wujie10558@gmail.com
user
wujie10
662a7078c9652518e153a737
Hefeng Wu
662a7078c9652518e153a738
Xuefeng Xiao
admin_assigned
2024-04-26T12:55:23.839Z
646b7f71df2609a541c1ab9f
/avatars/48b82e5fd9b06f41ff825507c36816cd.svg
Xuefeng Xiao
user
xiaoxuefeng
662a7078c9652518e153a739
Liang Lin
ID-Aligner: Enhancing Identity-Preserving Text-to-Image Generation with
  Reward Feedback Learning
The rapid development of diffusion models has triggered diverse applications.
Identity-preserving text-to-image generation (ID-T2I) particularly has received
significant attention due to its wide range of application scenarios like AI
portrait and advertising. While existing ID-T2I methods have demonstrated
impressive results, several key challenges remain: (1) It is hard to maintain
the identity characteristics of reference portraits accurately, (2) The
generated images lack aesthetic appeal especially while enforcing identity
retention, and (3) There is a limitation that cannot be compatible with
LoRA-based and Adapter-based methods simultaneously. To address these issues,
we present ID-Aligner, a general feedback learning framework to
enhance ID-T2I performance. To resolve identity features lost, we introduce
identity consistency reward fine-tuning to utilize the feedback from face
detection and recognition models to improve generated identity preservation.
Furthermore, we propose identity aesthetic reward fine-tuning leveraging
rewards from human-annotated preference data and automatically constructed
feedback on character structure generation to provide aesthetic tuning signals.
Thanks to its universal feedback fine-tuning framework, our method can be
readily applied to both LoRA and Adapter models, achieving consistent
performance gains. Extensive experiments on SD1.5 and SDXL diffusion models
validate the effectiveness of our approach. Project Page:
\url{https://idaligner.github.io/}
ID-Aligner enhances identity-preserving text-to-image generation through feedback learning, addressing challenges in identity retention, aesthetic appeal, and compatibility with parameter-efficient fine-tuning methods.
diffusion models identity-preserving text-to-image generation ID-T2I identity consistency reward fine-tuning identity aesthetic reward fine-tuning feedback learning LoRA Adapter SD1.5 SDXL