Learning Continuous 3D Words for Text-to-Image Generation
Current controls over diffusion models (e.g., through text or ControlNet) for
image generation fall short in recognizing abstract, continuous attributes like
illumination direction or non-rigid shape change. In this paper, we present an
approach for allowing users of text-to-image models to have fine-grained
control of several attributes in an image. We do this by engineering special
sets of input tokens that can be transformed in a continuous manner -- we call
them Continuous 3D Words. These attributes can, for example, be represented as
sliders and applied jointly with text prompts for fine-grained control over
image generation. Given only a single mesh and a rendering engine, we show that
our approach can be adopted to provide continuous user control over several
3D-aware attributes, including time-of-day illumination, bird wing orientation,
dollyzoom effect, and object poses. Our method is capable of conditioning image
creation with multiple Continuous 3D Words and text descriptions simultaneously
while adding no overhead to the generative process. Project Page:
https://ttchengab.github.io/continuous_3d_words
65cc2b53ac07b2595b264f95
Ta-Ying Cheng
claimed_verified
2024-02-15T08:28:53.410Z
6442b024e255a338677d78fa
/avatars/516769c6c3bc7a0d416924181c6866dc.svg
Ta-Ying Cheng
user
chengtim
65cc2b53ac07b2595b264f96
Matheus Gadelha
admin_assigned
2024-02-14T12:52:23.719Z
6315a9c466062476ae1fa0c7
/avatars/c59749be7ecee4858c59889fea61c88a.svg
Matheus
user
matheusgadelha
65cc2b53ac07b2595b264f97
Thibault Groueix
claimed_verified
2024-02-15T08:28:56.673Z
6340aad6b8b51e0098d7db14
/avatars/d3af16ef309dbd6fbe2fea27db68332f.svg
groueix
user
groueix
65cc2b53ac07b2595b264f98
Matthew Fisher
65cc2b53ac07b2595b264f99
Radomir Mech
65cc2b53ac07b2595b264f9a
Andrew Markham
admin_assigned
2024-02-14T12:52:56.940Z
63fca1aa9df782461c7c6367
/avatars/d61de110a4e87da80a5ca5820921773a.svg
Andrew Markham
user
markham
65cc2b53ac07b2595b264f9b
Niki Trigoni
Learning Continuous 3D Words for Text-to-Image Generation
Current controls over diffusion models (e.g., through text or ControlNet) for
image generation fall short in recognizing abstract, continuous attributes like
illumination direction or non-rigid shape change. In this paper, we present an
approach for allowing users of text-to-image models to have fine-grained
control of several attributes in an image. We do this by engineering special
sets of input tokens that can be transformed in a continuous manner -- we call
them Continuous 3D Words. These attributes can, for example, be represented as
sliders and applied jointly with text prompts for fine-grained control over
image generation. Given only a single mesh and a rendering engine, we show that
our approach can be adopted to provide continuous user control over several
3D-aware attributes, including time-of-day illumination, bird wing orientation,
dollyzoom effect, and object poses. Our method is capable of conditioning image
creation with multiple Continuous 3D Words and text descriptions simultaneously
while adding no overhead to the generative process. Project Page:
https://ttchengab.github.io/continuous_3d_words
Continuous 3D Words enable fine-grained control of 3D attributes in text-to-image models, allowing users to adjust image generation through continuous sliders alongside text prompts.
Continuous 3D Words text-to-image models multidimensional control time-of-day illumination bird wing orientation dollyzoom effect object poses text prompts rendering engine