StableSemantics: A Synthetic Language-Vision Dataset of Semantic
  Representations in Naturalistic Images
Understanding the semantics of visual scenes is a fundamental challenge in
Computer Vision. A key aspect of this challenge is that objects sharing similar
semantic meanings or functions can exhibit striking visual differences, making
accurate identification and categorization difficult. Recent advancements in
text-to-image frameworks have led to models that implicitly capture natural
scene statistics. These frameworks account for the visual variability of
objects, as well as complex object co-occurrences and sources of noise such as
diverse lighting conditions. By leveraging large-scale datasets and
cross-attention conditioning, these models generate detailed and contextually
rich scene representations. This capability opens new avenues for improving
object recognition and scene understanding in varied and challenging
environments. Our work presents StableSemantics, a dataset comprising 224
thousand human-curated prompts, processed natural language captions, over 2
million synthetic images, and 10 million attention maps corresponding to
individual noun chunks. We explicitly leverage human-generated prompts that
correspond to visually interesting stable diffusion generations, provide 10
generations per phrase, and extract cross-attention maps for each image. We
explore the semantic distribution of generated images, examine the distribution
of objects within images, and benchmark captioning and open vocabulary
segmentation methods on our data. To the best of our knowledge, we are the
first to release a diffusion dataset with semantic attributions. We expect our
proposed dataset to catalyze advances in visual semantic understanding and
provide a foundation for developing more sophisticated and effective visual
models. Website: https://stablesemantics.github.io/StableSemantics
6675d0a88794bbd33ffb1adb
Rushikesh Zawar
6675d0a88794bbd33ffb1adc
Shaurya Dewan
6675d0a88794bbd33ffb1add
Andrew F. Luo
extracted_confirmed
2024-06-21T19:15:43.344Z
64b6ce23dbbd1f2cdb624d56
/avatars/022738ce8f76fa9545b3e363d7264b53.svg
Andrew Luo
user
aluo-x
6675d0a88794bbd33ffb1ade
Margaret M. Henderson
6675d0a88794bbd33ffb1adf
Michael J. Tarr
extracted_pending
2024-06-21T19:12:44.405Z
649c2cdc92525b3ac9fe4f98
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/E4QRz0o81ROM4GyYo-xZN.jpeg
Michael J Tarr
user
Academic-Mercenary
6675d0a88794bbd33ffb1ae0
Leila Wehbe
StableSemantics: A Synthetic Language-Vision Dataset of Semantic
  Representations in Naturalistic Images
Understanding the semantics of visual scenes is a fundamental challenge in
Computer Vision. A key aspect of this challenge is that objects sharing similar
semantic meanings or functions can exhibit striking visual differences, making
accurate identification and categorization difficult. Recent advancements in
text-to-image frameworks have led to models that implicitly capture natural
scene statistics. These frameworks account for the visual variability of
objects, as well as complex object co-occurrences and sources of noise such as
diverse lighting conditions. By leveraging large-scale datasets and
cross-attention conditioning, these models generate detailed and contextually
rich scene representations. This capability opens new avenues for improving
object recognition and scene understanding in varied and challenging
environments. Our work presents StableSemantics, a dataset comprising 224
thousand human-curated prompts, processed natural language captions, over 2
million synthetic images, and 10 million attention maps corresponding to
individual noun chunks. We explicitly leverage human-generated prompts that
correspond to visually interesting stable diffusion generations, provide 10
generations per phrase, and extract cross-attention maps for each image. We
explore the semantic distribution of generated images, examine the distribution
of objects within images, and benchmark captioning and open vocabulary
segmentation methods on our data. To the best of our knowledge, we are the
first to release a diffusion dataset with semantic attributions. We expect our
proposed dataset to catalyze advances in visual semantic understanding and
provide a foundation for developing more sophisticated and effective visual
models. Website: https://stablesemantics.github.io/StableSemantics
StableSemantics is a dataset that includes prompts, captions, synthetic images, and attention maps to advance visual semantic understanding by benchmarking captioning and segmentation methods.
text-to-image frameworks diffusion models semantic meanings object co-occurrences cross-attention conditioning scene representations object recognition scene understanding semantic distribution open vocabulary segmentation diffusion dataset semantic attributions