Generating Compositional Scenes via Text-to-image RGBA Instance
  Generation
Text-to-image diffusion generative models can generate high quality images at
the cost of tedious prompt engineering. Controllability can be improved by
introducing layout conditioning, however existing methods lack layout editing
ability and fine-grained control over object attributes. The concept of
multi-layer generation holds great potential to address these limitations,
however generating image instances concurrently to scene composition limits
control over fine-grained object attributes, relative positioning in 3D space
and scene manipulation abilities. In this work, we propose a novel multi-stage
generation paradigm that is designed for fine-grained control, flexibility and
interactivity. To ensure control over instance attributes, we devise a novel
training paradigm to adapt a diffusion model to generate isolated scene
components as RGBA images with transparency information. To build complex
images, we employ these pre-generated instances and introduce a multi-layer
composite generation process that smoothly assembles components in realistic
scenes. Our experiments show that our RGBA diffusion model is capable of
generating diverse and high quality instances with precise control over object
attributes. Through multi-layer composition, we demonstrate that our approach
allows to build and manipulate images from highly complex prompts with
fine-grained control over object appearance and location, granting a higher
degree of control than competing methods.
673fc0c4bf2e9c35b2f51502
Alessandro Fontanella
claimed_verified
2024-11-22T10:02:04.397Z
673f2840be306dfff3d306df
/avatars/6d8f9fcaf1413e699afd82d62d4c5291.svg
Alessandro Fontanella
user
a-fontanella
673fc0c4bf2e9c35b2f51503
Petru-Daniel Tudosiu
admin_assigned
2024-11-22T13:22:23.915Z
63d44162ff1384ce6c636dce
https://cdn-avatars.huggingface.co/v1/production/uploads/1674854731168-noauth.jpeg
Petru-Daniel Tudosiu
user
danieltudosiu
673fc0c4bf2e9c35b2f51504
Yongxin Yang
admin_assigned
2024-11-22T13:22:30.375Z
62b6bdaadd998a8b1e3df69b
/avatars/c845b5fdc487430eedf2519675a2b455.svg
Yongxin Yang
user
HalleyX
673fc0c4bf2e9c35b2f51505
Shifeng Zhang
673fc0c4bf2e9c35b2f51506
Sarah Parisot
Generating Compositional Scenes via Text-to-image RGBA Instance
  Generation
Text-to-image diffusion generative models can generate high quality images at
the cost of tedious prompt engineering. Controllability can be improved by
introducing layout conditioning, however existing methods lack layout editing
ability and fine-grained control over object attributes. The concept of
multi-layer generation holds great potential to address these limitations,
however generating image instances concurrently to scene composition limits
control over fine-grained object attributes, relative positioning in 3D space
and scene manipulation abilities. In this work, we propose a novel multi-stage
generation paradigm that is designed for fine-grained control, flexibility and
interactivity. To ensure control over instance attributes, we devise a novel
training paradigm to adapt a diffusion model to generate isolated scene
components as RGBA images with transparency information. To build complex
images, we employ these pre-generated instances and introduce a multi-layer
composite generation process that smoothly assembles components in realistic
scenes. Our experiments show that our RGBA diffusion model is capable of
generating diverse and high quality instances with precise control over object
attributes. Through multi-layer composition, we demonstrate that our approach
allows to build and manipulate images from highly complex prompts with
fine-grained control over object appearance and location, granting a higher
degree of control than competing methods.
A multi-stage diffusion model generates high-quality images with fine-grained control over object attributes and scene composition through a transparent RGBA diffusion model and multi-layer composition.
diffusion generative models prompt engineering layout conditioning multi-layer generation multi-stage generation RGBA images composite generation scene components object attributes scene manipulation precise control object appearance location