Getting it Right: Improving Spatial Consistency in Text-to-Image Models
One of the key shortcomings in current text-to-image (T2I) models is their
inability to consistently generate images which faithfully follow the spatial
relationships specified in the text prompt. In this paper, we offer a
comprehensive investigation of this limitation, while also developing datasets
and methods that achieve state-of-the-art performance. First, we find that
current vision-language datasets do not represent spatial relationships well
enough; to alleviate this bottleneck, we create SPRIGHT, the first
spatially-focused, large scale dataset, by re-captioning 6 million images from
4 widely used vision datasets. Through a 3-fold evaluation and analysis
pipeline, we find that SPRIGHT largely improves upon existing datasets in
capturing spatial relationships. To demonstrate its efficacy, we leverage only
~0.25% of SPRIGHT and achieve a 22% improvement in generating spatially
accurate images while also improving the FID and CMMD scores. Secondly, we find
that training on images containing a large number of objects results in
substantial improvements in spatial consistency. Notably, we attain
state-of-the-art on T2I-CompBench with a spatial score of 0.2133, by
fine-tuning on <500 images. Finally, through a set of controlled experiments
and ablations, we document multiple findings that we believe will enhance the
understanding of factors that affect spatial consistency in text-to-image
models. We publicly release our dataset and model to foster further research in
this area.
660b8580cafce26b3c53f6f3
Agneet Chatterjee
extracted_confirmed
2024-04-02T06:49:43.337Z
6320c537a023aad6a7680c8b
/avatars/057dc492b8f756b83f12ced0b74fae65.svg
Agneet Chatterjee
user
agneet
660b8580cafce26b3c53f6f4
Gabriela Ben Melech Stan
claimed_verified
2024-04-03T09:27:03.353Z
63174cfddc97a974718f373a
/avatars/f5a1d5d90abf8aafe8864bc4f1b076cc.svg
gabriela ben melech
user
gabsm
660b8580cafce26b3c53f6f5
Estelle Aflalo
claimed_verified
2024-12-18T09:55:08.001Z
64575266900b28f7841090e5
/avatars/13dc0a370bb24271653c352897a2673e.svg
Estelle Aflalo
user
estellea
660b8580cafce26b3c53f6f6
Sayak Paul
claimed_verified
2024-04-02T11:02:54.848Z
5f7fbd813e94f16a85448745
https://cdn-avatars.huggingface.co/v1/production/uploads/1649681653581-5f7fbd813e94f16a85448745.jpeg
Sayak Paul
user
sayakpaul
660b8580cafce26b3c53f6f7
Dhruba Ghosh
claimed_verified
2024-04-02T19:35:04.166Z
638683b581fe8c678a39e80e
/avatars/4124b185c1abc6dfd30e1b9b529941f8.svg
Dhruba Ghosh
user
djghosh
660b8580cafce26b3c53f6f8
Tejas Gokhale
660b8580cafce26b3c53f6f9
Ludwig Schmidt
660b8580cafce26b3c53f6fa
Hannaneh Hajishirzi
660b8580cafce26b3c53f6fb
Vasudev Lal
660b8580cafce26b3c53f6fc
Chitta Baral
660b8580cafce26b3c53f6fd
Yezhou Yang
claimed_verified
2024-04-04T07:49:51.954Z
6486a850903d5878eb98110f
https://cdn-avatars.huggingface.co/v1/production/uploads/6486a850903d5878eb98110f/ETSE52Fdq5kVPzy4R5tGt.jpeg
'YZ' Yezhou Yang
user
AiWithoutBorders
Getting it Right: Improving Spatial Consistency in Text-to-Image Models
One of the key shortcomings in current text-to-image (T2I) models is their
inability to consistently generate images which faithfully follow the spatial
relationships specified in the text prompt. In this paper, we offer a
comprehensive investigation of this limitation, while also developing datasets
and methods that achieve state-of-the-art performance. First, we find that
current vision-language datasets do not represent spatial relationships well
enough; to alleviate this bottleneck, we create SPRIGHT, the first
spatially-focused, large scale dataset, by re-captioning 6 million images from
4 widely used vision datasets. Through a 3-fold evaluation and analysis
pipeline, we find that SPRIGHT largely improves upon existing datasets in
capturing spatial relationships. To demonstrate its efficacy, we leverage only
~0.25% of SPRIGHT and achieve a 22% improvement in generating spatially
accurate images while also improving the FID and CMMD scores. Secondly, we find
that training on images containing a large number of objects results in
substantial improvements in spatial consistency. Notably, we attain
state-of-the-art on T2I-CompBench with a spatial score of 0.2133, by
fine-tuning on <500 images. Finally, through a set of controlled experiments
and ablations, we document multiple findings that we believe will enhance the
understanding of factors that affect spatial consistency in text-to-image
models. We publicly release our dataset and model to foster further research in
this area.
The paper introduces SPRIGHT, a dataset focused on spatial relationships, and fine-tuning methods that significantly improve spatial consistency in text-to-image models.
text-to-image SPRIGHT spatial relationships vision-language datasets spatially-focused FID CMMD T2I-CompBench spatial score fine-tuning