ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free
  Domain Adaptation
Large-scale Pre-Training Vision-Language Model such as CLIP has demonstrated
outstanding performance in zero-shot classification, e.g. achieving 76.3% top-1
accuracy on ImageNet without seeing any example, which leads to potential
benefits to many tasks that have no labeled data. However, while applying CLIP
to a downstream target domain, the presence of visual and text domain gaps and
cross-modality misalignment can greatly impact the model performance. To
address such challenges, we propose ReCLIP, the first source-free domain
adaptation method for vision-language models, which does not require any source
data or target labeled data. ReCLIP first learns a projection space to mitigate
the misaligned visual-text embeddings and learns pseudo labels, and then
deploys cross-modality self-training with the pseudo labels, to update visual
and text encoders, refine labels and reduce domain gaps and misalignments
iteratively. With extensive experiments, we demonstrate ReCLIP reduces the
average error rate of CLIP from 30.17% to 25.06% on 22 image classification
benchmarks.
64d30aacaca4293a673dff47
Hu. Xuefeng
64d30aacaca4293a673dff48
Zhang. Ke
64d30aacaca4293a673dff49
Xia. Lu
64d30aacaca4293a673dff4a
Chen. Albert
64d30aacaca4293a673dff4b
Luo. Jiajia
64d30aacaca4293a673dff4c
Sun. Yuyin
64d30aacaca4293a673dff4d
Wang. Ken
64d30aacaca4293a673dff4e
Qiao. Nan
64d30aacaca4293a673dff4f
Zeng. Xiao
64d30aacaca4293a673dff50
Sun. Min
64d30aacaca4293a673dff51
Kuo. Cheng-Hao
64d30aacaca4293a673dff52
Nevatia. Ram
ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free
  Domain Adaptation
Large-scale Pre-Training Vision-Language Model such as CLIP has demonstrated
outstanding performance in zero-shot classification, e.g. achieving 76.3% top-1
accuracy on ImageNet without seeing any example, which leads to potential
benefits to many tasks that have no labeled data. However, while applying CLIP
to a downstream target domain, the presence of visual and text domain gaps and
cross-modality misalignment can greatly impact the model performance. To
address such challenges, we propose ReCLIP, the first source-free domain
adaptation method for vision-language models, which does not require any source
data or target labeled data. ReCLIP first learns a projection space to mitigate
the misaligned visual-text embeddings and learns pseudo labels, and then
deploys cross-modality self-training with the pseudo labels, to update visual
and text encoders, refine labels and reduce domain gaps and misalignments
iteratively. With extensive experiments, we demonstrate ReCLIP reduces the
average error rate of CLIP from 30.17% to 25.06% on 22 image classification
benchmarks.
ReCLIP, a source-free domain adaptation method for vision-language models, reduces the average error rate of CLIP by learning a projection space and deploying cross-modality self-training with pseudo labels.
vision-language models CLIP zero-shot classification domain gaps cross-modality misalignment source-free domain adaptation projection space pseudo labels cross-modality self-training visual encoders text encoders