LooseControl: Lifting ControlNet for Generalized Depth Conditioning
We present LooseControl to allow generalized depth conditioning for
diffusion-based image generation. ControlNet, the SOTA for depth-conditioned
image generation, produces remarkable results but relies on having access to
detailed depth maps for guidance. Creating such exact depth maps, in many
scenarios, is challenging. This paper introduces a generalized version of depth
conditioning that enables many new content-creation workflows. Specifically, we
allow (C1) scene boundary control for loosely specifying scenes with only
boundary conditions, and (C2) 3D box control for specifying layout locations of
the target objects rather than the exact shape and appearance of the objects.
Using LooseControl, along with text guidance, users can create complex
environments (e.g., rooms, street views, etc.) by specifying only scene
boundaries and locations of primary objects. Further, we provide two editing
mechanisms to refine the results: (E1) 3D box editing enables the user to
refine images by changing, adding, or removing boxes while freezing the style
of the image. This yields minimal changes apart from changes induced by the
edited boxes. (E2) Attribute editing proposes possible editing directions to
change one particular aspect of the scene, such as the overall object density
or a particular object. Extensive tests and comparisons with baselines
demonstrate the generality of our method. We believe that LooseControl can
become an important design tool for easily creating complex environments and be
extended to other forms of guidance channels. Code and more information are
available at https://shariqfarooq123.github.io/loose-control/ .
65712af9f3853f99bc43355d
Shariq Farooq Bhat
admin_assigned
2023-12-07T12:16:28.744Z
63fcae4f987f631186e50fba
/avatars/8ac9dff40d1b483b4f2f4e4cbd3f9088.svg
Shariq Farooq Bhat
user
shariqfarooq
65712af9f3853f99bc43355e
Niloy J. Mitra
65712af9f3853f99bc43355f
Peter Wonka
LooseControl: Lifting ControlNet for Generalized Depth Conditioning
We present LooseControl to allow generalized depth conditioning for
diffusion-based image generation. ControlNet, the SOTA for depth-conditioned
image generation, produces remarkable results but relies on having access to
detailed depth maps for guidance. Creating such exact depth maps, in many
scenarios, is challenging. This paper introduces a generalized version of depth
conditioning that enables many new content-creation workflows. Specifically, we
allow (C1) scene boundary control for loosely specifying scenes with only
boundary conditions, and (C2) 3D box control for specifying layout locations of
the target objects rather than the exact shape and appearance of the objects.
Using LooseControl, along with text guidance, users can create complex
environments (e.g., rooms, street views, etc.) by specifying only scene
boundaries and locations of primary objects. Further, we provide two editing
mechanisms to refine the results: (E1) 3D box editing enables the user to
refine images by changing, adding, or removing boxes while freezing the style
of the image. This yields minimal changes apart from changes induced by the
edited boxes. (E2) Attribute editing proposes possible editing directions to
change one particular aspect of the scene, such as the overall object density
or a particular object. Extensive tests and comparisons with baselines
demonstrate the generality of our method. We believe that LooseControl can
become an important design tool for easily creating complex environments and be
extended to other forms of guidance channels. Code and more information are
available at https://shariqfarooq123.github.io/loose-control/ .
LooseControl enables generalized depth conditioning for diffusion-based image generation using scene boundaries and 3D box control, enhancing flexibility and ease of use for creating complex environments.
diffusion-based image generation depth conditioning ControlNet scene boundary control 3D box control 3D box editing attribute editing object density