IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance
Ensuring precise multimodal alignment between diffusion-generated images and
input prompts has been a long-standing challenge. Earlier works finetune
diffusion weight using high-quality preference data, which tends to be limited
and difficult to scale up. Recent editing-based methods further refine local
regions of generated images but may compromise overall image quality. In this
work, we propose Implicit Multimodal Guidance (IMG), a novel
re-generation-based multimodal alignment framework that requires no extra data
or editing operations. Specifically, given a generated image and its prompt,
IMG a) utilizes a multimodal large language model (MLLM) to identify
misalignments; b) introduces an Implicit Aligner that manipulates diffusion
conditioning features to reduce misalignments and enable re-generation; and c)
formulates the re-alignment goal into a trainable objective, namely Iteratively
Updated Preference Objective. Extensive qualitative and quantitative
evaluations on SDXL, SDXL-DPO, and FLUX show that IMG outperforms existing
alignment methods. Furthermore, IMG acts as a flexible plug-and-play adapter,
seamlessly enhancing prior finetuning-based alignment methods. Our code will be
available at https://github.com/SHI-Labs/IMG-Multimodal-Diffusion-Alignment.
68dc9cd34159d1f2418f9b8e
Jiayi Guo
claimed_verified
2025-10-03T12:54:54.636Z
6417c111dce1e4c0229ec182
/avatars/de65da1f02b136b1af57862d9d0c48f2.svg
Jiayi Guo
user
JiayiGuo821
68dc9cd34159d1f2418f9b8f
Chuanhao Yan
68dc9cd34159d1f2418f9b90
Xingqian Xu
68dc9cd34159d1f2418f9b91
Yulin Wang
68dc9cd34159d1f2418f9b92
Kai Wang
68dc9cd34159d1f2418f9b93
Gao Huang
68dc9cd34159d1f2418f9b94
Humphrey Shi
IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance
Ensuring precise multimodal alignment between diffusion-generated images and
input prompts has been a long-standing challenge. Earlier works finetune
diffusion weight using high-quality preference data, which tends to be limited
and difficult to scale up. Recent editing-based methods further refine local
regions of generated images but may compromise overall image quality. In this
work, we propose Implicit Multimodal Guidance (IMG), a novel
re-generation-based multimodal alignment framework that requires no extra data
or editing operations. Specifically, given a generated image and its prompt,
IMG a) utilizes a multimodal large language model (MLLM) to identify
misalignments; b) introduces an Implicit Aligner that manipulates diffusion
conditioning features to reduce misalignments and enable re-generation; and c)
formulates the re-alignment goal into a trainable objective, namely Iteratively
Updated Preference Objective. Extensive qualitative and quantitative
evaluations on SDXL, SDXL-DPO, and FLUX show that IMG outperforms existing
alignment methods. Furthermore, IMG acts as a flexible plug-and-play adapter,
seamlessly enhancing prior finetuning-based alignment methods. Our code will be
available at https://github.com/SHI-Labs/IMG-Multimodal-Diffusion-Alignment.
Implicit Multimodal Guidance (IMG) enhances multimodal alignment between diffusion-generated images and prompts without additional data or editing, outperforming existing methods.
diffusion-generated images multimodal alignment diffusion weight preference data editing-based methods Implicit Multimodal Guidance multimodal large language model Implicit Aligner diffusion conditioning features Iteratively Updated Preference Objective SDXL SDXL-DPO FLUX