Emergent Correspondence from Image Diffusion
Finding correspondences between images is a fundamental problem in computer
vision. In this paper, we show that correspondence emerges in image diffusion
models without any explicit supervision. We propose a simple strategy to
extract this implicit knowledge out of diffusion networks as image features,
namely DIffusion FeaTures (DIFT), and use them to establish correspondences
between real images. Without any additional fine-tuning or supervision on the
task-specific data or annotations, DIFT is able to outperform both
weakly-supervised methods and competitive off-the-shelf features in identifying
semantic, geometric, and temporal correspondences. Particularly for semantic
correspondence, DIFT from Stable Diffusion is able to outperform DINO and
OpenCLIP by 19 and 14 accuracy points respectively on the challenging SPair-71k
benchmark. It even outperforms the state-of-the-art supervised methods on 9 out
of 18 categories while remaining on par for the overall performance. Project
page: https://diffusionfeatures.github.io
647fe3bcb82adfa7cc197ba0
Luming Tang
claimed_verified
2023-06-09T15:34:52.635Z
63e495a99db5da2dc1f679df
/avatars/e3b3f7702a5c8fd21a246272940e45b8.svg
Luming Tang
user
lt453
647fe3bcb82adfa7cc197ba1
Menglin Jia
claimed_verified
2023-06-15T09:17:44.863Z
648a4681fcda61143190645e
/avatars/270bca51ef4d8453920713e9a33c46b3.svg
Menglin Jia
user
kmnp
647fe3bcb82adfa7cc197ba2
Qianqian Wang
647fe3bcb82adfa7cc197ba3
Cheng Perng Phoo
admin_assigned
2023-06-12T14:04:41.144Z
64020b3582bbdfe4b7c44edd
/avatars/27c1d3eba6e89b4a56d1e171bffa8b80.svg
Cheng Perng Phoo
user
cpphoo
647fe3bcb82adfa7cc197ba4
Bharath Hariharan
Emergent Correspondence from Image Diffusion
Finding correspondences between images is a fundamental problem in computer
vision. In this paper, we show that correspondence emerges in image diffusion
models without any explicit supervision. We propose a simple strategy to
extract this implicit knowledge out of diffusion networks as image features,
namely DIffusion FeaTures (DIFT), and use them to establish correspondences
between real images. Without any additional fine-tuning or supervision on the
task-specific data or annotations, DIFT is able to outperform both
weakly-supervised methods and competitive off-the-shelf features in identifying
semantic, geometric, and temporal correspondences. Particularly for semantic
correspondence, DIFT from Stable Diffusion is able to outperform DINO and
OpenCLIP by 19 and 14 accuracy points respectively on the challenging SPair-71k
benchmark. It even outperforms the state-of-the-art supervised methods on 9 out
of 18 categories while remaining on par for the overall performance. Project
page: https://diffusionfeatures.github.io
Image diffusion models extract implicit features (DIFT) that outperform supervised and weakly-supervised methods in establishing semantic, geometric, and temporal correspondences between images.
diffusion models DIffusion FeaTures (DIFT) semantic correspondence geometric correspondence temporal correspondence Stable Diffusion DINO OpenCLIP SPair-71k benchmark