Diffusion Models as Data Mining Tools
This paper demonstrates how to use generative models trained for image
synthesis as tools for visual data mining. Our insight is that since
contemporary generative models learn an accurate representation of their
training data, we can use them to summarize the data by mining for visual
patterns. Concretely, we show that after finetuning conditional diffusion
models to synthesize images from a specific dataset, we can use these models to
define a typicality measure on that dataset. This measure assesses how typical
visual elements are for different data labels, such as geographic location,
time stamps, semantic labels, or even the presence of a disease. This
analysis-by-synthesis approach to data mining has two key advantages. First, it
scales much better than traditional correspondence-based approaches since it
does not require explicitly comparing all pairs of visual elements. Second,
while most previous works on visual data mining focus on a single dataset, our
approach works on diverse datasets in terms of content and scale, including a
historical car dataset, a historical face dataset, a large worldwide
street-view dataset, and an even larger scene dataset. Furthermore, our
approach allows for translating visual elements across class labels and
analyzing consistent changes.
66b2d253fa7069c7f1eef4df
Ioannis Siglidis
66b2d253fa7069c7f1eef4e0
Aleksander Holynski
admin_assigned
2024-08-08T12:24:00.571Z
62f6e2792e53c2efd33faa92
/avatars/85d94b4022577747b8d2d10a82c2f3c7.svg
Aleksander Holynski
user
holynski
66b2d253fa7069c7f1eef4e1
Alexei A. Efros
66b2d253fa7069c7f1eef4e2
Mathieu Aubry
66b2d253fa7069c7f1eef4e3
Shiry Ginosar
Diffusion Models as Data Mining Tools
This paper demonstrates how to use generative models trained for image
synthesis as tools for visual data mining. Our insight is that since
contemporary generative models learn an accurate representation of their
training data, we can use them to summarize the data by mining for visual
patterns. Concretely, we show that after finetuning conditional diffusion
models to synthesize images from a specific dataset, we can use these models to
define a typicality measure on that dataset. This measure assesses how typical
visual elements are for different data labels, such as geographic location,
time stamps, semantic labels, or even the presence of a disease. This
analysis-by-synthesis approach to data mining has two key advantages. First, it
scales much better than traditional correspondence-based approaches since it
does not require explicitly comparing all pairs of visual elements. Second,
while most previous works on visual data mining focus on a single dataset, our
approach works on diverse datasets in terms of content and scale, including a
historical car dataset, a historical face dataset, a large worldwide
street-view dataset, and an even larger scene dataset. Furthermore, our
approach allows for translating visual elements across class labels and
analyzing consistent changes.
The approach uses fine-tuned conditional diffusion models to define a typicality measure for visual data elements across various labels and datasets, improving scalability and versatility in visual data mining.
generative models image synthesis visual data mining conditional diffusion models typicality measure geographic location time stamps semantic labels disease presence analysis-by-synthesis scalability historical car dataset historical face dataset worldwide street-view dataset scene dataset visual element translation consistent changes