Disentangling Diffusion Features for Detecting Inconsistencies in Subject-Driven Generation
Abdelrahman Eldesokey, Aleksandar Cvejić, Bernard Ghanem, Peter Wonka
We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models, enabling the detection of visually inconsistent regions in subject-driven image generation. While diffusion model backbones are known to encode semantically rich features, they should also contain visual features that capture appearance to support their image synthesis capabilities. However, isolating these visual features is non-trivial due to the absence of datasets with annotated visual correspondences.To address this, we design an automated dataset generation pipeline that produces image pairs with annotated semantic and visual correspondences based on existing subject-driven datasets. Using this dataset, we propose an architecture to disentangle semantic and visual features in a contrastive manner. We further propose a metric that leverages the disentangled features to quantify and localize inconsistencies in subject-driven generation.Experiments show that our approach significantly outperforms global feature-based metrics such as CLIP and DINO, as well as Vision-Language Models, in capturing visual inconsistencies. To the best of our knowledge, this is the first approach that enables both quantification and spatial localization of inconsistency in subject-driven image generation, offering a valuable tool for advancing the task.