Controllable Text-to-Image Generation with GPT-4
Current text-to-image generation models often struggle to follow textual
instructions, especially the ones requiring spatial reasoning. On the other
hand, Large Language Models (LLMs), such as GPT-4, have shown remarkable
precision in generating code snippets for sketching out text inputs
graphically, e.g., via TikZ. In this work, we introduce Control-GPT to guide
the diffusion-based text-to-image pipelines with programmatic sketches
generated by GPT-4, enhancing their abilities for instruction following.
Control-GPT works by querying GPT-4 to write TikZ code, and the generated
sketches are used as references alongside the text instructions for diffusion
models (e.g., ControlNet) to generate photo-realistic images. One major
challenge to training our pipeline is the lack of a dataset containing aligned
text, images, and sketches. We address the issue by converting instance masks
in existing datasets into polygons to mimic the sketches used at test time. As
a result, Control-GPT greatly boosts the controllability of image generation.
It establishes a new state-of-art on the spatial arrangement and object
positioning generation and enhances users' control of object positions, sizes,
etc., nearly doubling the accuracy of prior models. Our work, as a first
attempt, shows the potential for employing LLMs to enhance the performance in
computer vision tasks.
64769a94288784ce9abc35b9
Tianjun Zhang
admin_assigned
2023-06-01T08:14:19.658Z
6374cd6b6ea8da14f8fef8dc
https://cdn-avatars.huggingface.co/v1/production/uploads/6374cd6b6ea8da14f8fef8dc/l13bg0tKDjCnUw3I895QZ.png
Tianjun Zhang
user
tianjunz
64769a94288784ce9abc35ba
Yi Zhang
64769a94288784ce9abc35bb
Vibhav Vineet
admin_assigned
2023-06-02T12:36:23.726Z
63c8527becdb7c9fdd9cacc6
/avatars/c8a3f5e1e5159ae5ead41bd9fc2b9b34.svg
Vibhav Vineet
user
vibhav-vineet
64769a94288784ce9abc35bc
Neel Joshi
64769a94288784ce9abc35bd
Xin Wang
claimed_verified
2024-04-09T18:39:41.291Z
64679a226192d39142245e5e
/avatars/05abee0b6317f100923936ca2099e9eb.svg
Xin Eric Wang
user
xw-eric
Controllable Text-to-Image Generation with GPT-4
Current text-to-image generation models often struggle to follow textual
instructions, especially the ones requiring spatial reasoning. On the other
hand, Large Language Models (LLMs), such as GPT-4, have shown remarkable
precision in generating code snippets for sketching out text inputs
graphically, e.g., via TikZ. In this work, we introduce Control-GPT to guide
the diffusion-based text-to-image pipelines with programmatic sketches
generated by GPT-4, enhancing their abilities for instruction following.
Control-GPT works by querying GPT-4 to write TikZ code, and the generated
sketches are used as references alongside the text instructions for diffusion
models (e.g., ControlNet) to generate photo-realistic images. One major
challenge to training our pipeline is the lack of a dataset containing aligned
text, images, and sketches. We address the issue by converting instance masks
in existing datasets into polygons to mimic the sketches used at test time. As
a result, Control-GPT greatly boosts the controllability of image generation.
It establishes a new state-of-art on the spatial arrangement and object
positioning generation and enhances users' control of object positions, sizes,
etc., nearly doubling the accuracy of prior models. Our work, as a first
attempt, shows the potential for employing LLMs to enhance the performance in
computer vision tasks.
Control-GPT enhances text-to-image generation by using GPT-4-generated TikZ sketches as guides, improving spatial arrangement and object positioning accuracy.
diffusion-based text-to-image ControlNet TikZ instance masks polygons graphical sketches Large Language Models LLMs control of image generation spatial reasoning object positioning text-to-image