DreamTeacher: Pretraining Image Backbones with Deep Generative Models
In this work, we introduce a self-supervised feature representation learning
framework DreamTeacher that utilizes generative networks for pre-training
downstream image backbones. We propose to distill knowledge from a trained
generative model into standard image backbones that have been well engineered
for specific perception tasks. We investigate two types of knowledge
distillation: 1) distilling learned generative features onto target image
backbones as an alternative to pretraining these backbones on large labeled
datasets such as ImageNet, and 2) distilling labels obtained from generative
networks with task heads onto logits of target backbones. We perform extensive
analyses on multiple generative models, dense prediction benchmarks, and
several pre-training regimes. We empirically find that our DreamTeacher
significantly outperforms existing self-supervised representation learning
approaches across the board. Unsupervised ImageNet pre-training with
DreamTeacher leads to significant improvements over ImageNet classification
pre-training on downstream datasets, showcasing generative models, and
diffusion generative models specifically, as a promising approach to
representation learning on large, diverse datasets without requiring manual
annotation.
64b4aceeffed52962ad7a971
Daiqing Li
admin_assigned
2023-07-17T11:09:16.227Z
633f44efc11d723b1809958b
/avatars/d70b067deabddae984c3637290b7de99.svg
Li
user
Daiqing
64b4aceeffed52962ad7a972
Huan Ling
claimed_verified
2023-07-18T06:44:12.036Z
64a1910bb14ab77f9e46497a
/avatars/5e6b63233274e89bf74d132eae03a374.svg
Huan
user
linghuan4545
64b4aceeffed52962ad7a973
Amlan Kar
admin_assigned
2023-07-17T11:09:44.598Z
619ebcc653061ce00477b08d
/avatars/f754e144accaacb8f18331d5d49a6a37.svg
Amlan Kar
user
amlankar
64b4aceeffed52962ad7a974
David Acuna
admin_assigned
2023-07-17T11:10:00.229Z
645abb43c4acfcf6640270e3
/avatars/e65f8de9b9fe878bd8e9106a802c6f21.svg
David Acuna
user
davidjesusacu
64b4aceeffed52962ad7a975
Seung Wook Kim
64b4aceeffed52962ad7a976
Karsten Kreis
64b4aceeffed52962ad7a977
Antonio Torralba
64b4aceeffed52962ad7a978
Sanja Fidler
DreamTeacher: Pretraining Image Backbones with Deep Generative Models
In this work, we introduce a self-supervised feature representation learning
framework DreamTeacher that utilizes generative networks for pre-training
downstream image backbones. We propose to distill knowledge from a trained
generative model into standard image backbones that have been well engineered
for specific perception tasks. We investigate two types of knowledge
distillation: 1) distilling learned generative features onto target image
backbones as an alternative to pretraining these backbones on large labeled
datasets such as ImageNet, and 2) distilling labels obtained from generative
networks with task heads onto logits of target backbones. We perform extensive
analyses on multiple generative models, dense prediction benchmarks, and
several pre-training regimes. We empirically find that our DreamTeacher
significantly outperforms existing self-supervised representation learning
approaches across the board. Unsupervised ImageNet pre-training with
DreamTeacher leads to significant improvements over ImageNet classification
pre-training on downstream datasets, showcasing generative models, and
diffusion generative models specifically, as a promising approach to
representation learning on large, diverse datasets without requiring manual
annotation.
DreamTeacher uses generative networks to distill knowledge into self-supervised image backbones, improving performance across various benchmarks and datasets without labeled data.
self-supervised feature representation DreamTeacher generative networks pre-training dense prediction knowledge distillation ImageNet diffusion generative models representation learning