StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual
  Representation Learners
We investigate the potential of learning visual representations using
synthetic images generated by text-to-image models. This is a natural question
in the light of the excellent performance of such models in generating
high-quality images. We consider specifically the Stable Diffusion, one of the
leading open source text-to-image models. We show that (1) when the generative
model is configured with proper classifier-free guidance scale, training
self-supervised methods on synthetic images can match or beat the real image
counterpart; (2) by treating the multiple images generated from the same text
prompt as positives for each other, we develop a multi-positive contrastive
learning method, which we call StableRep. With solely synthetic images, the
representations learned by StableRep surpass the performance of representations
learned by SimCLR and CLIP using the same set of text prompts and corresponding
real images, on large scale datasets. When we further add language supervision,
StableRep trained with 20M synthetic images achieves better accuracy than CLIP
trained with 50M real images.
64797ff450ff7001631a4451
Yonglong Tian
admin_assigned
2023-06-02T07:30:44.118Z
64793086ac77bb7603842456
/avatars/5891292138d372a9cc063f86c08b8174.svg
Yonglong Tian
user
tyl5566
64797ff450ff7001631a4452
Lijie Fan
64797ff450ff7001631a4453
Phillip Isola
admin_assigned
2023-06-02T07:30:06.506Z
623cb1f3ca87989bbf797eee
/avatars/13fbd5b123b8676561751f820317600c.svg
Phillip Isola
user
phillipi
64797ff450ff7001631a4454
Huiwen Chang
64797ff450ff7001631a4455
Dilip Krishnan
StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual
  Representation Learners
We investigate the potential of learning visual representations using
synthetic images generated by text-to-image models. This is a natural question
in the light of the excellent performance of such models in generating
high-quality images. We consider specifically the Stable Diffusion, one of the
leading open source text-to-image models. We show that (1) when the generative
model is configured with proper classifier-free guidance scale, training
self-supervised methods on synthetic images can match or beat the real image
counterpart; (2) by treating the multiple images generated from the same text
prompt as positives for each other, we develop a multi-positive contrastive
learning method, which we call StableRep. With solely synthetic images, the
representations learned by StableRep surpass the performance of representations
learned by SimCLR and CLIP using the same set of text prompts and corresponding
real images, on large scale datasets. When we further add language supervision,
StableRep trained with 20M synthetic images achieves better accuracy than CLIP
trained with 50M real images.
Training self-supervised methods on synthetic images generated by a text-to-image model like Stable Diffusion can yield high-performing visual representations matching or exceeding those trained on real images.
text-to-image models Stable Diffusion classifier-free guidance scale self-supervised methods multi-positive contrastive learning StableRep SimCLR CLIP large scale datasets language supervision