MoDE: CLIP Data Experts via Clustering
The success of contrastive language-image pretraining (CLIP) relies on the
supervision from the pairing between images and captions, which tends to be
noisy in web-crawled data. We present Mixture of Data Experts (MoDE) and learn
a system of CLIP data experts via clustering. Each data expert is trained on
one data cluster, being less sensitive to false negative noises in other
clusters. At inference time, we ensemble their outputs by applying weights
determined through the correlation between task metadata and cluster
conditions. To estimate the correlation precisely, the samples in one cluster
should be semantically similar, but the number of data experts should still be
reasonable for training and inference. As such, we consider the ontology in
human language and propose to use fine-grained cluster centers to represent
each data expert at a coarse-grained level. Experimental studies show that four
CLIP data experts on ViT-B/16 outperform the ViT-L/14 by OpenAI CLIP and
OpenCLIP on zero-shot image classification but with less (<35\%) training
cost. Meanwhile, MoDE can train all data expert asynchronously and can flexibly
include new data experts. The code is available at
https://github.com/facebookresearch/MetaCLIP/tree/main/mode.
662a9cab16435ebd17bf5e5f
Jiawei Ma
admin_assigned
2024-04-26T12:10:10.328Z
653b41de251b430ac072def1
/avatars/cfefeca4e242f7e32c3db4fb7d9f812a.svg
Jiawei Ma
user
Phoenix1128
662a9cab16435ebd17bf5e60
Po-Yao Huang
662a9cab16435ebd17bf5e61
Saining Xie
admin_assigned
2024-04-26T12:10:34.373Z
6596422646624a86ff3b3bda
/avatars/216e12b77e45ac5f1fa20932f5745411.svg
Saining Xie
user
sainx
662a9cab16435ebd17bf5e62
Shang-Wen Li
claimed_verified
2024-04-28T19:30:16.890Z
64e3a7b8e12618b261fff364
/avatars/860a5011f150537eb923f76d354f1bc5.svg
Shang-Wen Daniel Li
user
swdanielli
662a9cab16435ebd17bf5e63
Luke Zettlemoyer
662a9cab16435ebd17bf5e64
Shih-Fu Chang
admin_assigned
2024-04-26T12:11:05.711Z
6616bd1fdb7e6d38160080c4
/avatars/ba927a1c97e090b61eea9a3e04750692.svg
Jeff
user
shihfu
662a9cab16435ebd17bf5e65
Wen-Tau Yih
662a9cab16435ebd17bf5e66
Hu Xu
MoDE: CLIP Data Experts via Clustering
The success of contrastive language-image pretraining (CLIP) relies on the
supervision from the pairing between images and captions, which tends to be
noisy in web-crawled data. We present Mixture of Data Experts (MoDE) and learn
a system of CLIP data experts via clustering. Each data expert is trained on
one data cluster, being less sensitive to false negative noises in other
clusters. At inference time, we ensemble their outputs by applying weights
determined through the correlation between task metadata and cluster
conditions. To estimate the correlation precisely, the samples in one cluster
should be semantically similar, but the number of data experts should still be
reasonable for training and inference. As such, we consider the ontology in
human language and propose to use fine-grained cluster centers to represent
each data expert at a coarse-grained level. Experimental studies show that four
CLIP data experts on ViT-B/16 outperform the ViT-L/14 by OpenAI CLIP and
OpenCLIP on zero-shot image classification but with less (<35\%) training
cost. Meanwhile, MoDE can train all data expert asynchronously and can flexibly
include new data experts. The code is available at
https://github.com/facebookresearch/MetaCLIP/tree/main/mode.
Mixture of Data Experts (MoDE) enhances CLIP's performance by clustering data into experts, reducing noise and improving zero-shot classification with lower training costs.
contrastive language-image pretraining CLIP supervision data experts clustering false negative noises ensemble task metadata cluster conditions ontology fine-grained cluster centers coarse-grained level zero-shot image classification training cost asynchronous training