Diffusion Classifiers Understand Compositionality, but Conditions Apply
Understanding visual scenes is fundamental to human intelligence. While
discriminative models have significantly advanced computer vision, they often
struggle with compositional understanding. In contrast, recent generative
text-to-image diffusion models excel at synthesizing complex scenes, suggesting
inherent compositional capabilities. Building on this, zero-shot diffusion
classifiers have been proposed to repurpose diffusion models for discriminative
tasks. While prior work offered promising results in discriminative
compositional scenarios, these results remain preliminary due to a small number
of benchmarks and a relatively shallow analysis of conditions under which the
models succeed. To address this, we present a comprehensive study of the
discriminative capabilities of diffusion classifiers on a wide range of
compositional tasks. Specifically, our study covers three diffusion models (SD
1.5, 2.0, and, for the first time, 3-m) spanning 10 datasets and over 30 tasks.
Further, we shed light on the role that target dataset domains play in
respective performance; to isolate the domain effects, we introduce a new
diagnostic benchmark Self-Bench comprised of images created by diffusion models
themselves. Finally, we explore the importance of timestep weighting and
uncover a relationship between domain gap and timestep sensitivity,
particularly for SD3-m. To sum up, diffusion classifiers understand
compositionality, but conditions apply! Code and dataset are available at
https://github.com/eugene6923/Diffusion-Classifiers-Compositionality.
68343fcab624d4711d816434
Yujin Jeong
claimed_verified
2025-05-26T13:18:42.290Z
63a14dd9b5515dccd425275e
/avatars/791342bd41112bc97b5f8164d761d259.svg
Jeong
user
eugene6923
68343fcab624d4711d816435
Arnas Uselis
claimed_verified
2025-05-26T11:59:06.709Z
6520898f7bf8cc2dd28b7a9c
/avatars/87a29ba95b71ee2dce18e97aa85e17a1.svg
Arnas Uselis
user
Gigglingface
68343fcab624d4711d816436
Seong Joon Oh
claimed_verified
2025-05-28T09:00:58.023Z
638a50450f10aa3064f03f23
/avatars/0c068458e42950c851758a238225c3a6.svg
Seong Joon Oh
user
coallaoh
68343fcab624d4711d816437
Anna Rohrbach
Diffusion Classifiers Understand Compositionality, but Conditions Apply
Understanding visual scenes is fundamental to human intelligence. While
discriminative models have significantly advanced computer vision, they often
struggle with compositional understanding. In contrast, recent generative
text-to-image diffusion models excel at synthesizing complex scenes, suggesting
inherent compositional capabilities. Building on this, zero-shot diffusion
classifiers have been proposed to repurpose diffusion models for discriminative
tasks. While prior work offered promising results in discriminative
compositional scenarios, these results remain preliminary due to a small number
of benchmarks and a relatively shallow analysis of conditions under which the
models succeed. To address this, we present a comprehensive study of the
discriminative capabilities of diffusion classifiers on a wide range of
compositional tasks. Specifically, our study covers three diffusion models (SD
1.5, 2.0, and, for the first time, 3-m) spanning 10 datasets and over 30 tasks.
Further, we shed light on the role that target dataset domains play in
respective performance; to isolate the domain effects, we introduce a new
diagnostic benchmark Self-Bench comprised of images created by diffusion models
themselves. Finally, we explore the importance of timestep weighting and
uncover a relationship between domain gap and timestep sensitivity,
particularly for SD3-m. To sum up, diffusion classifiers understand
compositionality, but conditions apply! Code and dataset are available at
https://github.com/eugene6923/Diffusion-Classifiers-Compositionality.
A study of diffusion classifiers across multiple datasets and tasks reveals their compositional understanding, highlighting domain-specific performance effects and timestep weighting importance.
discriminative models generative text-to-image diffusion models zero-shot diffusion classifiers compositional tasks SD 1.5 SD 2.0 SD3-m Self-Bench diagnostic benchmark timestep weighting domain gap timestep sensitivity