A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style
  Models on Dense Captions
Curation methods for massive vision-language datasets trade off between
dataset size and quality. However, even the highest quality of available
curated captions are far too short to capture the rich visual detail in an
image. To show the value of dense and highly-aligned image-text pairs, we
collect the Densely Captioned Images (DCI) dataset, containing 8012 natural
images human-annotated with mask-aligned descriptions averaging above 1000
words each. With precise and reliable captions associated with specific parts
of an image, we can evaluate vision-language models' (VLMs) understanding of
image content with a novel task that matches each caption with its
corresponding subcrop. As current models are often limited to 77 text tokens,
we also introduce a summarized version (sDCI) in which each caption length is
limited. We show that modern techniques that make progress on standard
benchmarks do not correspond with significant improvement on our sDCI based
benchmark. Lastly, we finetune CLIP using sDCI and show significant
improvements over the baseline despite a small training set. By releasing the
first human annotated dense image captioning dataset, we hope to enable the
development of new benchmarks or fine-tuning recipes for the next generation of
VLMs to come.
657bc69bddc32bef8bb96956
Jack Urbanek
admin_assigned
2023-12-15T10:45:30.439Z
631106397680dc699b1b1ca1
/avatars/2c6f31b5328309cf4b42262bbb1c6372.svg
Jack Urbanek
user
JackUrb
657bc69bddc32bef8bb96957
Florian Bordes
admin_assigned
2023-12-15T10:45:41.654Z
63166336894404e2506b8811
/avatars/ec51a1dd86511127cf83afd4bf7c0f52.svg
Florian Bordes
user
Fbordes
657bc69bddc32bef8bb96958
Pietro Astolfi
admin_assigned
2023-12-15T10:45:48.973Z
646388dfd4d34b01f45d87c3
/avatars/a66bf45e780d0a82d00035f16c24d4c8.svg
Pietro Astolfi
user
pietroastolfi
657bc69bddc32bef8bb96959
Mary Williamson
657bc69bddc32bef8bb9695a
Vasu Sharma
claimed_verified
2025-01-10T08:22:27.973Z
65492730a3d1682f79f3ab7a
/avatars/fc62605bdf10947dab393305c51ace96.svg
Vasu Sharma
user
vasusharma55
657bc69bddc32bef8bb9695b
Adriana Romero-Soriano
A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style
  Models on Dense Captions
Curation methods for massive vision-language datasets trade off between
dataset size and quality. However, even the highest quality of available
curated captions are far too short to capture the rich visual detail in an
image. To show the value of dense and highly-aligned image-text pairs, we
collect the Densely Captioned Images (DCI) dataset, containing 8012 natural
images human-annotated with mask-aligned descriptions averaging above 1000
words each. With precise and reliable captions associated with specific parts
of an image, we can evaluate vision-language models' (VLMs) understanding of
image content with a novel task that matches each caption with its
corresponding subcrop. As current models are often limited to 77 text tokens,
we also introduce a summarized version (sDCI) in which each caption length is
limited. We show that modern techniques that make progress on standard
benchmarks do not correspond with significant improvement on our sDCI based
benchmark. Lastly, we finetune CLIP using sDCI and show significant
improvements over the baseline despite a small training set. By releasing the
first human annotated dense image captioning dataset, we hope to enable the
development of new benchmarks or fine-tuning recipes for the next generation of
VLMs to come.
A human-annotated dataset with dense captions significantly improves the evaluation and fine-tuning of vision-language models beyond standard benchmarks.
Vision-language models (VLMs) mask-aligned descriptions subcrop matching text tokens densel captioning CLIP fine-tuning