Scaling Language-Free Visual Representation Learning
Visual Self-Supervised Learning (SSL) currently underperforms Contrastive
Language-Image Pretraining (CLIP) in multimodal settings such as Visual
Question Answering (VQA). This multimodal gap is often attributed to the
semantics introduced by language supervision, even though visual SSL and CLIP
models are often trained on different data. In this work, we ask the question:
"Do visual self-supervised approaches lag behind CLIP due to the lack of
language supervision, or differences in the training data?" We study this
question by training both visual SSL and CLIP models on the same MetaCLIP data,
and leveraging VQA as a diverse testbed for vision encoders. In this controlled
setup, visual SSL models scale better than CLIP models in terms of data and
model capacity, and visual SSL performance does not saturate even after scaling
up to 7B parameters. Consequently, we observe visual SSL methods achieve
CLIP-level performance on a wide range of VQA and classic vision benchmarks.
These findings demonstrate that pure visual SSL can match language-supervised
visual pretraining at scale, opening new opportunities for vision-centric
representation learning.
67ecebc51f669fb5591616cd
David Fan
admin_assigned
2025-04-02T11:18:13.746Z
66e9df8c38f5bae5a91785ca
/avatars/6f49429fc34553903309872ec6f122b8.svg
David Fan
user
davidfan97
67ecebc51f669fb5591616ce
Shengbang Tong
admin_assigned
2025-04-02T11:18:31.173Z
6374cbb7255276f3a22b4b35
/avatars/7cf1bbb83447441e5fa2e1e4fcf7617b.svg
Peter Tong
user
tsbpp
67ecebc51f669fb5591616cf
Jiachen Zhu
admin_assigned
2025-04-02T11:54:32.112Z
6552126dd8a8835b66653767
/avatars/0b1dad9ebaeada8f5e7ebe453123960b.svg
Jiachen Zhu
user
JiachenZhu
67ecebc51f669fb5591616d0
Koustuv Sinha
admin_assigned
2025-04-02T11:54:39.909Z
622b93067b9143726fbedc37
https://cdn-avatars.huggingface.co/v1/production/uploads/1647023505150-622b93067b9143726fbedc37.jpeg
Koustuv Sinha
user
koustuvs
67ecebc51f669fb5591616d1
Zhuang Liu
admin_assigned
2025-04-02T11:55:01.118Z
647d3a9383cdb93baf76dce2
/avatars/56efca912a66ef9d4f2f3679aac31bae.svg
Zhuang Liu
user
liuzhuang13
67ecebc51f669fb5591616d2
Xinlei Chen
admin_assigned
2025-04-02T11:55:07.362Z
63e58e3a006a775275e59e41
/avatars/75262a35b27a2ae1939df9118120d99e.svg
Xinlei Chen
user
endernewton
67ecebc51f669fb5591616d3
Michael Rabbat
67ecebc51f669fb5591616d4
Nicolas Ballas
67ecebc51f669fb5591616d5
Yann LeCun
admin_assigned
2025-04-02T11:55:22.112Z
64ed0b8c2203a126eb1a5b9a
/avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
Yann LeCun
user
ylecun
67ecebc51f669fb5591616d6
Amir Bar
admin_assigned
2025-04-02T11:55:47.451Z
632bad473690fb57e70c9d42
/avatars/8c09bc614a7e62a7bd30c4cbb97d4122.svg
Amir Bar
user
amirbar1
67ecebc51f669fb5591616d7
Saining Xie
admin_assigned
2025-04-02T11:55:54.586Z
6596422646624a86ff3b3bda
/avatars/216e12b77e45ac5f1fa20932f5745411.svg
Saining Xie
user
sainx
Scaling Language-Free Visual Representation Learning
Visual Self-Supervised Learning (SSL) currently underperforms Contrastive
Language-Image Pretraining (CLIP) in multimodal settings such as Visual
Question Answering (VQA). This multimodal gap is often attributed to the
semantics introduced by language supervision, even though visual SSL and CLIP
models are often trained on different data. In this work, we ask the question:
"Do visual self-supervised approaches lag behind CLIP due to the lack of
language supervision, or differences in the training data?" We study this
question by training both visual SSL and CLIP models on the same MetaCLIP data,
and leveraging VQA as a diverse testbed for vision encoders. In this controlled
setup, visual SSL models scale better than CLIP models in terms of data and
model capacity, and visual SSL performance does not saturate even after scaling
up to 7B parameters. Consequently, we observe visual SSL methods achieve
CLIP-level performance on a wide range of VQA and classic vision benchmarks.
These findings demonstrate that pure visual SSL can match language-supervised
visual pretraining at scale, opening new opportunities for vision-centric
representation learning.
Visual self-supervised learning matches language-supervised visual pretraining performance on VQA and vision benchmarks when both are trained on the same dataset and scaled appropriately.
visual self-supervised learning contrastive language-image pretraining CLIP visual question answering VQA MetaCLIP vision encoders data and model capacity