InstanceGen: Image Generation with Instance-level Instructions
Despite rapid advancements in the capabilities of generative models,
pretrained text-to-image models still struggle in capturing the semantics
conveyed by complex prompts that compound multiple objects and instance-level
attributes. Consequently, we are witnessing growing interests in integrating
additional structural constraints, typically in the form of coarse bounding
boxes, to better guide the generation process in such challenging cases. In
this work, we take the idea of structural guidance a step further by making the
observation that contemporary image generation models can directly provide a
plausible fine-grained structural initialization. We propose a technique that
couples this image-based structural guidance with LLM-based instance-level
instructions, yielding output images that adhere to all parts of the text
prompt, including object counts, instance-level attributes, and spatial
relations between instances.
6828f2e4159fce854b16f5fa
Etai Sella
extracted_pending
2025-05-17T20:34:50.662Z
63b3e4b4b7fec0adf64ded69
/avatars/55804581c263c1d68b6fc59a1b0a3df7.svg
Etai  Sella
user
etaisella
6828f2e4159fce854b16f5fb
Yanir Kleiman
6828f2e4159fce854b16f5fc
Hadar Averbuch-Elor
InstanceGen: Image Generation with Instance-level Instructions
Despite rapid advancements in the capabilities of generative models,
pretrained text-to-image models still struggle in capturing the semantics
conveyed by complex prompts that compound multiple objects and instance-level
attributes. Consequently, we are witnessing growing interests in integrating
additional structural constraints, typically in the form of coarse bounding
boxes, to better guide the generation process in such challenging cases. In
this work, we take the idea of structural guidance a step further by making the
observation that contemporary image generation models can directly provide a
plausible fine-grained structural initialization. We propose a technique that
couples this image-based structural guidance with LLM-based instance-level
instructions, yielding output images that adhere to all parts of the text
prompt, including object counts, instance-level attributes, and spatial
relations between instances.
The proposed technique combines image-based structural guidance with LLM-based instructions to align generated images with complex, detailed text prompts.
pretrained text-to-image models semantic prompts coarse bounding boxes fine-grained structural initialization instance-level instructions object counts instance-level attributes spatial relations