Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in
  Subject-Driven Generation
We propose a novel approach for disentangling visual and semantic features
from the backbones of pre-trained diffusion models, enabling visual
correspondence in a manner analogous to the well-established semantic
correspondence. While diffusion model backbones are known to encode
semantically rich features, they must also contain visual features to support
their image synthesis capabilities. However, isolating these visual features is
challenging due to the absence of annotated datasets. To address this, we
introduce an automated pipeline that constructs image pairs with annotated
semantic and visual correspondences based on existing subject-driven image
generation datasets, and design a contrastive architecture to separate the two
feature types. Leveraging the disentangled representations, we propose a new
metric, Visual Semantic Matching (VSM), that quantifies visual inconsistencies
in subject-driven image generation. Empirical results show that our approach
outperforms global feature-based metrics such as CLIP, DINO, and
vision--language models in quantifying visual inconsistencies while also
enabling spatial localization of inconsistent regions. To our knowledge, this
is the first method that supports both quantification and localization of
inconsistencies in subject-driven generation, offering a valuable tool for
advancing this task. Project
Page:https://abdo-eldesokey.github.io/mind-the-glitch/
68da1cfe0177a6054b013bc5
Abdelrahman Eldesokey
claimed_verified
2025-09-29T11:26:25.220Z
6577f3bacfb2207f11e847bb
/avatars/825998cfebc47d8106f633be5ad10964.svg
Abdelrahman Eldesokey
user
abdo-eldesokey
68da1cfe0177a6054b013bc6
Aleksandar Cvejic
68da1cfe0177a6054b013bc7
Bernard Ghanem
68da1cfe0177a6054b013bc8
Peter Wonka
Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in
  Subject-Driven Generation
We propose a novel approach for disentangling visual and semantic features
from the backbones of pre-trained diffusion models, enabling visual
correspondence in a manner analogous to the well-established semantic
correspondence. While diffusion model backbones are known to encode
semantically rich features, they must also contain visual features to support
their image synthesis capabilities. However, isolating these visual features is
challenging due to the absence of annotated datasets. To address this, we
introduce an automated pipeline that constructs image pairs with annotated
semantic and visual correspondences based on existing subject-driven image
generation datasets, and design a contrastive architecture to separate the two
feature types. Leveraging the disentangled representations, we propose a new
metric, Visual Semantic Matching (VSM), that quantifies visual inconsistencies
in subject-driven image generation. Empirical results show that our approach
outperforms global feature-based metrics such as CLIP, DINO, and
vision--language models in quantifying visual inconsistencies while also
enabling spatial localization of inconsistent regions. To our knowledge, this
is the first method that supports both quantification and localization of
inconsistencies in subject-driven generation, offering a valuable tool for
advancing this task. Project
Page:https://abdo-eldesokey.github.io/mind-the-glitch/
A novel method disentangles visual and semantic features from diffusion model backbones to quantify and localize visual inconsistencies in subject-driven image generation.
diffusion models visual features semantic features image pairs contrastive architecture Visual Semantic Matching (VSM) CLIP DINO vision--language models subject-driven image generation