GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesis
Text-to-image (T2I) generation has seen significant progress with diffusion
models, enabling generation of photo-realistic images from text prompts.
Despite this progress, existing methods still face challenges in following
complex text prompts, especially those requiring compositional and multi-step
reasoning. Given such complex instructions, SOTA models often make mistakes in
faithfully modeling object attributes, and relationships among them. In this
work, we present an alternate paradigm for T2I synthesis, decomposing the task
of complex multi-step generation into three steps, (a) Generate: we first
generate an image using existing diffusion models (b) Plan: we make use of
Multi-Modal LLMs (MLLMs) to identify the mistakes in the generated image
expressed in terms of individual objects and their properties, and produce a
sequence of corrective steps required in the form of an edit-plan. (c) Edit: we
make use of an existing text-guided image editing models to sequentially
execute our edit-plan over the generated image to get the desired image which
is faithful to the original instruction. Our approach derives its strength from
the fact that it is modular in nature, is training free, and can be applied
over any combination of image generation and editing models. As an added
contribution, we also develop a model capable of compositional editing, which
further helps improve the overall accuracy of our proposed approach. Our method
flexibly trades inference time compute with performance on compositional text
prompts. We perform extensive experimental evaluation across 3 benchmarks and
10 T2I models including DALLE-3 and the latest -- SD-3.5-Large. Our approach
not only improves the performance of the SOTA models, by upto 3 points, it also
reduces the performance gap between weaker and stronger models.
https://dair-iitd.github.io/GraPE/{https://dair-iitd.github.io/GraPE/}
67599c0672fd1d07c3b42832
Ashish Goswami
claimed_verified
2024-12-11T14:07:56.054Z
60c8304a672c96a317b85508
https://cdn-avatars.huggingface.co/v1/production/uploads/60c8304a672c96a317b85508/AfvceQ_1ZtDK7XU3qAmvb.jpeg
Ashish Goswami
user
aggr8
67599c0672fd1d07c3b42833
Satyam Kumar Modi
67599c0672fd1d07c3b42834
Santhosh Rishi Deshineni
67599c0672fd1d07c3b42835
Harman Singh
claimed_verified
2025-05-06T08:37:25.288Z
631c4048e65cd7f686d2ad4f
/avatars/49902bcd18da8079bacf312edc386982.svg
Harman Singh
user
harman
67599c0672fd1d07c3b42836
Prathosh A. P
67599c0672fd1d07c3b42837
Parag Singla
GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesis
Text-to-image (T2I) generation has seen significant progress with diffusion
models, enabling generation of photo-realistic images from text prompts.
Despite this progress, existing methods still face challenges in following
complex text prompts, especially those requiring compositional and multi-step
reasoning. Given such complex instructions, SOTA models often make mistakes in
faithfully modeling object attributes, and relationships among them. In this
work, we present an alternate paradigm for T2I synthesis, decomposing the task
of complex multi-step generation into three steps, (a) Generate: we first
generate an image using existing diffusion models (b) Plan: we make use of
Multi-Modal LLMs (MLLMs) to identify the mistakes in the generated image
expressed in terms of individual objects and their properties, and produce a
sequence of corrective steps required in the form of an edit-plan. (c) Edit: we
make use of an existing text-guided image editing models to sequentially
execute our edit-plan over the generated image to get the desired image which
is faithful to the original instruction. Our approach derives its strength from
the fact that it is modular in nature, is training free, and can be applied
over any combination of image generation and editing models. As an added
contribution, we also develop a model capable of compositional editing, which
further helps improve the overall accuracy of our proposed approach. Our method
flexibly trades inference time compute with performance on compositional text
prompts. We perform extensive experimental evaluation across 3 benchmarks and
10 T2I models including DALLE-3 and the latest -- SD-3.5-Large. Our approach
not only improves the performance of the SOTA models, by upto 3 points, it also
reduces the performance gap between weaker and stronger models.
https://dair-iitd.github.io/GraPE/{https://dair-iitd.github.io/GraPE/}
The proposed method decomposes text-to-image generation into generation, planning, and editing steps using diffusion models, multi-modal LLMs, and text-guided image editing, improving performance on complex text prompts and bridging performance gaps between different models.
diffusion models multi-modal LLMs text-to-image generation text-guided image editing edit-plan compositional editing DALLE-3 SD-3.5-Large