Describing Differences in Image Sets with Natural Language
How do two sets of images differ? Discerning set-level differences is crucial
for understanding model behaviors and analyzing datasets, yet manually sifting
through thousands of images is impractical. To aid in this discovery process,
we explore the task of automatically describing the differences between two
sets of images, which we term Set Difference Captioning. This task
takes in image sets D_A and D_B, and outputs a description that is more
often true on D_A than D_B. We outline a two-stage approach that first
proposes candidate difference descriptions from image sets and then re-ranks
the candidates by checking how well they can differentiate the two sets. We
introduce VisDiff, which first captions the images and prompts a language model
to propose candidate descriptions, then re-ranks these descriptions using CLIP.
To evaluate VisDiff, we collect VisDiffBench, a dataset with 187 paired image
sets with ground truth difference descriptions. We apply VisDiff to various
domains, such as comparing datasets (e.g., ImageNet vs. ImageNetV2), comparing
classification models (e.g., zero-shot CLIP vs. supervised ResNet), summarizing
model failure modes (supervised ResNet), characterizing differences between
generative models (e.g., StableDiffusionV1 and V2), and discovering what makes
images memorable. Using VisDiff, we are able to find interesting and previously
unknown differences in datasets and models, demonstrating its utility in
revealing nuanced insights.
656fd7dc6871b53dba94535a
Lisa Dunlap
extracted_confirmed
2023-12-06T02:13:36.794Z
63f7ccb8ad35fabd915171a8
/avatars/786bc609e6305730f0f2025ae5c27f82.svg
Lisa Dunlap
user
lisabdunlap
656fd7dc6871b53dba94535b
Yuhui Zhang
extracted_confirmed
2023-12-06T02:59:27.624Z
62da55164398e21bf7f0e292
https://cdn-avatars.huggingface.co/v1/production/uploads/62da55164398e21bf7f0e292/xjKkG8IA2IZZqCdjApSh3.jpeg
Yuhui Zhang
user
yuhuizhang
656fd7dc6871b53dba94535c
Xiaohan Wang
claimed_verified
2024-12-16T09:45:02.498Z
65703fab7f50602340d23704
/avatars/324c45f5fba9cd8c38a89b30427c06b4.svg
Xiaohan Wang
user
nicholswang
656fd7dc6871b53dba94535d
Ruiqi Zhong
admin_assigned
2023-12-06T11:51:56.831Z
5eae5de74e876668a0c37779
/avatars/cf04ab4832c6565ed085dec12fcae188.svg
Ruiqi Zhong
user
ruiqi-zhong
656fd7dc6871b53dba94535e
Trevor Darrell
admin_assigned
2023-12-06T11:52:06.354Z
64cbdf02f103036e23d1c7f3
/avatars/496069463900dea20929b57381182d39.svg
Trevor Darrell
user
trevordarrell
656fd7dc6871b53dba94535f
Jacob Steinhardt
656fd7dc6871b53dba945360
Joseph E. Gonzalez
extracted_pending
2023-12-06T02:09:35.113Z
645d2e8401f4eaab2a0878ce
/avatars/1273c5fb607b4b622a746a42692fa632.svg
Joseph E. Gonzalez
user
ProfJoeyG
656fd7dc6871b53dba945361
Serena Yeung-Levy
extracted_pending
2025-01-07T02:05:14.400Z
677c8b2e92550a07fcad0f50
/avatars/2be26e8f25e98cfe5b1d227ee0409cd0.svg
Serena Yeung-Levy
user
yeunglevy
Describing Differences in Image Sets with Natural Language
How do two sets of images differ? Discerning set-level differences is crucial
for understanding model behaviors and analyzing datasets, yet manually sifting
through thousands of images is impractical. To aid in this discovery process,
we explore the task of automatically describing the differences between two
sets of images, which we term Set Difference Captioning. This task
takes in image sets D_A and D_B, and outputs a description that is more
often true on D_A than D_B. We outline a two-stage approach that first
proposes candidate difference descriptions from image sets and then re-ranks
the candidates by checking how well they can differentiate the two sets. We
introduce VisDiff, which first captions the images and prompts a language model
to propose candidate descriptions, then re-ranks these descriptions using CLIP.
To evaluate VisDiff, we collect VisDiffBench, a dataset with 187 paired image
sets with ground truth difference descriptions. We apply VisDiff to various
domains, such as comparing datasets (e.g., ImageNet vs. ImageNetV2), comparing
classification models (e.g., zero-shot CLIP vs. supervised ResNet), summarizing
model failure modes (supervised ResNet), characterizing differences between
generative models (e.g., StableDiffusionV1 and V2), and discovering what makes
images memorable. Using VisDiff, we are able to find interesting and previously
unknown differences in datasets and models, demonstrating its utility in
revealing nuanced insights.
VisDiff automatically generates descriptions of differences between two sets of images, aiding in the analysis of datasets and model behaviors across various applications.
Set Difference Captioning VisDiff candidate descriptions language model CLIP VisDiffBench zero-shot CLIP supervised ResNet StableDiffusion model failure modes generative models image memorability