Existing approaches for controlling text-to-image diffusion models, while
powerful, do not allow for explicit 3D object-centric control, such as precise
control of object orientation. In this work, we address the problem of
multi-object orientation control in text-to-image diffusion models. This
enables the generation of diverse multi-object scenes with precise orientation
control for each object. The key idea is to condition the diffusion model with
a set of orientation-aware compass tokens, one for each object, along
with text tokens. A light-weight encoder network predicts these compass tokens
taking object orientation as the input. The model is trained on a synthetic
dataset of procedurally generated scenes, each containing one or two 3D assets
on a plain background. However, direct training this framework results in poor
orientation control as well as leads to entanglement among objects. To mitigate
this, we intervene in the generation process and constrain the cross-attention
maps of each compass token to its corresponding object regions. The trained
model is able to achieve precise orientation control for a) complex objects not
seen during training and b) multi-object scenes with more than two objects,
indicating strong generalization capabilities. Further, when combined with
personalization methods, our method precisely controls the orientation of the
new object in diverse contexts. Our method achieves state-of-the-art
orientation control and text alignment, quantified with extensive evaluations
and a user study.
