Grounded Text-to-Image Synthesis with Attention Refocusing
Driven by scalable diffusion models trained on large-scale paired text-image
datasets, text-to-image synthesis methods have shown compelling results.
However, these models still fail to precisely follow the text prompt when
multiple objects, attributes, and spatial compositions are involved in the
prompt. In this paper, we identify the potential reasons in both the
cross-attention and self-attention layers of the diffusion model. We propose
two novel losses to refocus the attention maps according to a given layout
during the sampling process. We perform comprehensive experiments on the
DrawBench and HRS benchmarks using layouts synthesized by Large Language
Models, showing that our proposed losses can be integrated easily and
effectively into existing text-to-image methods and consistently improve their
alignment between the generated images and the text prompts.
6482a54fd3973485034deac0
Quynh Phung
6482a54fd3973485034deac1
Songwei Ge
admin_assigned
2023-06-12T13:30:53.093Z
631cdc696e623740a78fd864
/avatars/f610873f6080453b655fb230d6f3b765.svg
Songwei Ge
user
songweig
6482a54fd3973485034deac2
Jia-Bin Huang
admin_assigned
2023-06-12T13:31:15.328Z
641c139b73296f7ee256970c
/avatars/5a2550d95e686640242840ad3bd0e680.svg
Jiabin Huang
user
YellowAddice
Grounded Text-to-Image Synthesis with Attention Refocusing
Driven by scalable diffusion models trained on large-scale paired text-image
datasets, text-to-image synthesis methods have shown compelling results.
However, these models still fail to precisely follow the text prompt when
multiple objects, attributes, and spatial compositions are involved in the
prompt. In this paper, we identify the potential reasons in both the
cross-attention and self-attention layers of the diffusion model. We propose
two novel losses to refocus the attention maps according to a given layout
during the sampling process. We perform comprehensive experiments on the
DrawBench and HRS benchmarks using layouts synthesized by Large Language
Models, showing that our proposed losses can be integrated easily and
effectively into existing text-to-image methods and consistently improve their
alignment between the generated images and the text prompts.
Two novel losses improve attention focus in diffusion models, enhancing text-to-image alignment on benchmarks.
diffusion models text-to-image synthesis cross-attention self-attention attention maps layout DrawBench HRS benchmarks Large Language Models