CleanDIFT: Diffusion Features without Noise
Internal features from large-scale pre-trained diffusion models have recently
been established as powerful semantic descriptors for a wide range of
downstream tasks. Works that use these features generally need to add noise to
images before passing them through the model to obtain the semantic features,
as the models do not offer the most useful features when given images with
little to no noise. We show that this noise has a critical impact on the
usefulness of these features that cannot be remedied by ensembling with
different random noises. We address this issue by introducing a lightweight,
unsupervised fine-tuning method that enables diffusion backbones to provide
high-quality, noise-free semantic features. We show that these features readily
outperform previous diffusion features by a wide margin in a wide variety of
extraction setups and downstream tasks, offering better performance than even
ensemble-based methods at a fraction of the cost.
67516be84af3ce83f1e7dc67
Nick Stracke
admin_assigned
2024-12-05T10:44:40.411Z
63a1c8d63c003e409311e3c3
/avatars/c6cd546874abd5ee68ed88eff5ee1aa8.svg
Nick Stracke
user
kliyer
67516be84af3ce83f1e7dc68
Stefan Andreas Baumann
admin_assigned
2024-12-05T10:45:04.279Z
6471c12a0c2b5fdaf1f07c45
/avatars/6783067212d24d1e716a8b4c64df61b4.svg
Stefan Baumann
user
stefan-baumann
67516be84af3ce83f1e7dc69
Kolja Bauer
admin_assigned
2024-12-05T10:45:11.300Z
65ae67bcbff3ae9f331c846a
/avatars/6f9572fbfdba105638112386cdc346fa.svg
Kolja Bauer
user
kolja-b
67516be84af3ce83f1e7dc6a
Frank Fundel
admin_assigned
2024-12-05T10:45:17.457Z
623dd9817d2f5b63ac1b73df
https://cdn-avatars.huggingface.co/v1/production/uploads/1648220555715-623dd9817d2f5b63ac1b73df.jpeg
Frank Fundel
user
FrankFundel
67516be84af3ce83f1e7dc6b
Björn Ommer
CleanDIFT: Diffusion Features without Noise
Internal features from large-scale pre-trained diffusion models have recently
been established as powerful semantic descriptors for a wide range of
downstream tasks. Works that use these features generally need to add noise to
images before passing them through the model to obtain the semantic features,
as the models do not offer the most useful features when given images with
little to no noise. We show that this noise has a critical impact on the
usefulness of these features that cannot be remedied by ensembling with
different random noises. We address this issue by introducing a lightweight,
unsupervised fine-tuning method that enables diffusion backbones to provide
high-quality, noise-free semantic features. We show that these features readily
outperform previous diffusion features by a wide margin in a wide variety of
extraction setups and downstream tasks, offering better performance than even
ensemble-based methods at a fraction of the cost.
A lightweight, unsupervised fine-tuning method improves the quality of noise-free semantic features from diffusion models, surpassing ensemble-based methods in various tasks.
diffusion models semantic descriptors downstream tasks noise diffusion backbones fine-tuning unsupervised learning ensemble-based methods