GrounDiT: Grounding Diffusion Transformers via Noisy Patch
  Transplantation
We introduce a novel training-free spatial grounding technique for
text-to-image generation using Diffusion Transformers (DiT). Spatial grounding
with bounding boxes has gained attention for its simplicity and versatility,
allowing for enhanced user control in image generation. However, prior
training-free approaches often rely on updating the noisy image during the
reverse diffusion process via backpropagation from custom loss functions, which
frequently struggle to provide precise control over individual bounding boxes.
In this work, we leverage the flexibility of the Transformer architecture,
demonstrating that DiT can generate noisy patches corresponding to each
bounding box, fully encoding the target object and allowing for fine-grained
control over each region. Our approach builds on an intriguing property of DiT,
which we refer to as semantic sharing. Due to semantic sharing, when a smaller
patch is jointly denoised alongside a generatable-size image, the two become
"semantic clones". Each patch is denoised in its own branch of the generation
process and then transplanted into the corresponding region of the original
noisy image at each timestep, resulting in robust spatial grounding for each
bounding box. In our experiments on the HRS and DrawBench benchmarks, we
achieve state-of-the-art performance compared to previous training-free spatial
grounding approaches.
672074e1ddbde77be9cfc529
Phillip Y. Lee
claimed_verified
2024-10-30T09:53:19.885Z
6342796a0875f2c99cfd313b
/avatars/98575092404c4197b20c929a6499a015.svg
Yuseung "Phillip" Lee
user
phillipinseoul
672074e1ddbde77be9cfc52a
Taehoon Yoon
672074e1ddbde77be9cfc52b
Minhyuk Sung
GrounDiT: Grounding Diffusion Transformers via Noisy Patch
  Transplantation
We introduce a novel training-free spatial grounding technique for
text-to-image generation using Diffusion Transformers (DiT). Spatial grounding
with bounding boxes has gained attention for its simplicity and versatility,
allowing for enhanced user control in image generation. However, prior
training-free approaches often rely on updating the noisy image during the
reverse diffusion process via backpropagation from custom loss functions, which
frequently struggle to provide precise control over individual bounding boxes.
In this work, we leverage the flexibility of the Transformer architecture,
demonstrating that DiT can generate noisy patches corresponding to each
bounding box, fully encoding the target object and allowing for fine-grained
control over each region. Our approach builds on an intriguing property of DiT,
which we refer to as semantic sharing. Due to semantic sharing, when a smaller
patch is jointly denoised alongside a generatable-size image, the two become
"semantic clones". Each patch is denoised in its own branch of the generation
process and then transplanted into the corresponding region of the original
noisy image at each timestep, resulting in robust spatial grounding for each
bounding box. In our experiments on the HRS and DrawBench benchmarks, we
achieve state-of-the-art performance compared to previous training-free spatial
grounding approaches.
A novel training-free method using Diffusion Transformers achieves state-of-the-art spatial grounding in text-to-image generation by generating and denoising patches corresponding to each bounding box.
Diffusion Transformers DiT spatial grounding bounding boxes reverse diffusion process backpropagation custom loss functions Transformer architecture semantic sharing semantic clones noise patches generatable-size image HRS benchmark DrawBench benchmark