Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD
  Generalization
Existing vision-language models exhibit strong generalization on a variety of
visual domains and tasks. However, such models mainly perform zero-shot
recognition in a closed-set manner, and thus struggle to handle open-domain
visual concepts by design. There are recent finetuning methods, such as prompt
learning, that not only study the discrimination between in-distribution (ID)
and out-of-distribution (OOD) samples, but also show some improvements in both
ID and OOD accuracies. In this paper, we first demonstrate that vision-language
models, after long enough finetuning but without proper regularization, tend to
overfit the known classes in the given dataset, with degraded performance on
unknown classes. Then we propose a novel approach OGEN to address this pitfall,
with the main focus on improving the OOD GENeralization of finetuned models.
Specifically, a class-conditional feature generator is introduced to synthesize
OOD features using just the class name of any unknown class. Such synthesized
features will provide useful knowledge about unknowns and help regularize the
decision boundary between ID and OOD data when optimized jointly. Equally
important is our adaptive self-distillation mechanism to regularize our feature
generation model during joint optimization, i.e., adaptively transferring
knowledge between model states to further prevent overfitting. Experiments
validate that our method yields convincing gains in OOD generalization
performance in different settings.
65b868edcda408e65ffacfeb
Yuhang Zang
admin_assigned
2024-01-30T10:05:53.449Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/kajwuVzd4pDucSPlwghxo.png
Yuhang Zang
user
yuhangzang
65b868edcda408e65ffacfec
Hanlin Goh
65b868edcda408e65ffacfed
Josh Susskind
claimed_verified
2024-11-22T16:30:51.561Z
6470d2247fd7ecdbd0ec3cc9
/avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
Joshua M. Susskind
user
jsusskind
65b868edcda408e65ffacfee
Chen Huang
Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD
  Generalization
Existing vision-language models exhibit strong generalization on a variety of
visual domains and tasks. However, such models mainly perform zero-shot
recognition in a closed-set manner, and thus struggle to handle open-domain
visual concepts by design. There are recent finetuning methods, such as prompt
learning, that not only study the discrimination between in-distribution (ID)
and out-of-distribution (OOD) samples, but also show some improvements in both
ID and OOD accuracies. In this paper, we first demonstrate that vision-language
models, after long enough finetuning but without proper regularization, tend to
overfit the known classes in the given dataset, with degraded performance on
unknown classes. Then we propose a novel approach OGEN to address this pitfall,
with the main focus on improving the OOD GENeralization of finetuned models.
Specifically, a class-conditional feature generator is introduced to synthesize
OOD features using just the class name of any unknown class. Such synthesized
features will provide useful knowledge about unknowns and help regularize the
decision boundary between ID and OOD data when optimized jointly. Equally
important is our adaptive self-distillation mechanism to regularize our feature
generation model during joint optimization, i.e., adaptively transferring
knowledge between model states to further prevent overfitting. Experiments
validate that our method yields convincing gains in OOD generalization
performance in different settings.
A new method, OGEN, improves out-of-distribution generalization in finetuned vision-language models by synthesizing features for unknown classes and using adaptive self-distillation.
vision-language models zero-shot recognition prompt learning in-distribution out-of-distribution finetuning class-conditional feature generator OOD generalization adaptive self-distillation