UniControl: A Unified Diffusion Model for Controllable Visual Generation
  In the Wild
Achieving machine autonomy and human control often represent divergent
objectives in the design of interactive AI systems. Visual generative
foundation models such as Stable Diffusion show promise in navigating these
goals, especially when prompted with arbitrary languages. However, they often
fall short in generating images with spatial, structural, or geometric
controls. The integration of such controls, which can accommodate various
visual conditions in a single unified model, remains an unaddressed challenge.
In response, we introduce UniControl, a new generative foundation model that
consolidates a wide array of controllable condition-to-image (C2I) tasks within
a singular framework, while still allowing for arbitrary language prompts.
UniControl enables pixel-level-precise image generation, where visual
conditions primarily influence the generated structures and language prompts
guide the style and context. To equip UniControl with the capacity to handle
diverse visual conditions, we augment pretrained text-to-image diffusion models
and introduce a task-aware HyperNet to modulate the diffusion models, enabling
the adaptation to different C2I tasks simultaneously. Trained on nine unique
C2I tasks, UniControl demonstrates impressive zero-shot generation abilities
with unseen visual conditions. Experimental results show that UniControl often
surpasses the performance of single-task-controlled methods of comparable model
sizes. This control versatility positions UniControl as a significant
advancement in the realm of controllable visual generation.
6466de79cdea48159aa5263f
Can Qin
claimed_verified
2023-05-19T19:04:33.320Z
62ccfc52bcaa438a5dbe34de
/avatars/b7534388b0da3b3c6d805c515c76d672.svg
Can Qin
user
Robert001
6466de79cdea48159aa52640
Shu Zhang
6466de79cdea48159aa52641
Ning Yu
claimed_verified
2023-05-19T19:04:36.728Z
6362bcbe8f43a912fc722969
https://cdn-avatars.huggingface.co/v1/production/uploads/6362bcbe8f43a912fc722969/ktl2ePfpOIseqlIbuldNa.png
Ning Yu
user
ningyu1991
6466de79cdea48159aa52642
Yihao Feng
claimed_verified
2023-05-19T19:03:57.933Z
6354d31106d707b3324a3a5a
/avatars/a662d176dfc4b809cd75aa726f24b74c.svg
Yihao Feng
user
yihaocs
6466de79cdea48159aa52643
Xinyi Yang
6466de79cdea48159aa52644
Yingbo Zhou
6466de79cdea48159aa52645
Huan Wang
6466de79cdea48159aa52646
Juan Carlos Niebles
claimed_verified
2024-02-01T08:45:19.907Z
65bb26b5d61b51a50805aba2
/avatars/966c8469f55776948a7cf42d98accff3.svg
Juan Carlos Niebles
user
niebles
6466de79cdea48159aa52647
Caiming Xiong
6466de79cdea48159aa52648
Silvio Savarese
6466de79cdea48159aa52649
Stefano Ermon
6466de79cdea48159aa5264a
Yun Fu
6466de79cdea48159aa5264b
Ran Xu
claimed_verified
2023-05-19T19:03:54.237Z
6465c4c863e7e09dd02e3e1b
/avatars/200b029184d2616f98296a2c212f0785.svg
Ran Xu
user
xurantju
UniControl: A Unified Diffusion Model for Controllable Visual Generation
  In the Wild
Achieving machine autonomy and human control often represent divergent
objectives in the design of interactive AI systems. Visual generative
foundation models such as Stable Diffusion show promise in navigating these
goals, especially when prompted with arbitrary languages. However, they often
fall short in generating images with spatial, structural, or geometric
controls. The integration of such controls, which can accommodate various
visual conditions in a single unified model, remains an unaddressed challenge.
In response, we introduce UniControl, a new generative foundation model that
consolidates a wide array of controllable condition-to-image (C2I) tasks within
a singular framework, while still allowing for arbitrary language prompts.
UniControl enables pixel-level-precise image generation, where visual
conditions primarily influence the generated structures and language prompts
guide the style and context. To equip UniControl with the capacity to handle
diverse visual conditions, we augment pretrained text-to-image diffusion models
and introduce a task-aware HyperNet to modulate the diffusion models, enabling
the adaptation to different C2I tasks simultaneously. Trained on nine unique
C2I tasks, UniControl demonstrates impressive zero-shot generation abilities
with unseen visual conditions. Experimental results show that UniControl often
surpasses the performance of single-task-controlled methods of comparable model
sizes. This control versatility positions UniControl as a significant
advancement in the realm of controllable visual generation.
UniControl, a generative foundation model, integrates diverse visual controls and arbitrary language prompts, enabling precise image generation across multiple condition-to-image tasks while surpassing single-task control methods.
Visual generative foundation models Stable Diffusion condition-to-image (C2I) tasks text-to-image diffusion models HyperNet zero-shot generation control versatility structural controls geometric controls spatial controls