Compass Control: Multi Object Orientation Control for Text-to-Image
  Generation
Existing approaches for controlling text-to-image diffusion models, while
powerful, do not allow for explicit 3D object-centric control, such as precise
control of object orientation. In this work, we address the problem of
multi-object orientation control in text-to-image diffusion models. This
enables the generation of diverse multi-object scenes with precise orientation
control for each object. The key idea is to condition the diffusion model with
a set of orientation-aware compass tokens, one for each object, along
with text tokens. A light-weight encoder network predicts these compass tokens
taking object orientation as the input. The model is trained on a synthetic
dataset of procedurally generated scenes, each containing one or two 3D assets
on a plain background. However, direct training this framework results in poor
orientation control as well as leads to entanglement among objects. To mitigate
this, we intervene in the generation process and constrain the cross-attention
maps of each compass token to its corresponding object regions. The trained
model is able to achieve precise orientation control for a) complex objects not
seen during training and b) multi-object scenes with more than two objects,
indicating strong generalization capabilities. Further, when combined with
personalization methods, our method precisely controls the orientation of the
new object in diverse contexts. Our method achieves state-of-the-art
orientation control and text alignment, quantified with extensive evaluations
and a user study.
67f8c9f028f4f60ceb4103e4
Rishubh Parihar
admin_assigned
2025-04-11T08:43:15.754Z
631af10b492c0c57a22ea3e2
https://cdn-avatars.huggingface.co/v1/production/uploads/1662710000523-noauth.png
Rishubh Parihar
user
RishubhPar
67f8c9f028f4f60ceb4103e5
Vaibhav Agrawal
claimed_verified
2025-04-17T08:05:25.600Z
67ab8a5cc8e2f51341bc52f4
/avatars/99ee27bb922d04d098da17db902b3700.svg
Vaibhav Agrawal
user
va1bhavagrawa1
67f8c9f028f4f60ceb4103e6
Sachidanand VS
admin_assigned
2025-04-11T08:43:46.850Z
64f56843ad81562b476ec05d
/avatars/fa90a685259f89bd1b447ba93359187e.svg
sachidanand vs
user
sachi1
67f8c9f028f4f60ceb4103e7
R. Venkatesh Babu
Compass Control: Multi Object Orientation Control for Text-to-Image
  Generation
Existing approaches for controlling text-to-image diffusion models, while
powerful, do not allow for explicit 3D object-centric control, such as precise
control of object orientation. In this work, we address the problem of
multi-object orientation control in text-to-image diffusion models. This
enables the generation of diverse multi-object scenes with precise orientation
control for each object. The key idea is to condition the diffusion model with
a set of orientation-aware compass tokens, one for each object, along
with text tokens. A light-weight encoder network predicts these compass tokens
taking object orientation as the input. The model is trained on a synthetic
dataset of procedurally generated scenes, each containing one or two 3D assets
on a plain background. However, direct training this framework results in poor
orientation control as well as leads to entanglement among objects. To mitigate
this, we intervene in the generation process and constrain the cross-attention
maps of each compass token to its corresponding object regions. The trained
model is able to achieve precise orientation control for a) complex objects not
seen during training and b) multi-object scenes with more than two objects,
indicating strong generalization capabilities. Further, when combined with
personalization methods, our method precisely controls the orientation of the
new object in diverse contexts. Our method achieves state-of-the-art
orientation control and text alignment, quantified with extensive evaluations
and a user study.
This work addresses precise 3D object orientation control in text-to-image diffusion models by introducing orientation-aware compass tokens and constraining cross-attention maps.
text-to-image diffusion models multi-object orientation control compass tokens light-weight encoder cross-attention maps orientation control procedural generation generalization capabilities personalization methods text alignment