VESSA: Video-based Efficient SSL Adaptation
Jesimon Barreto, Carlos Caetano, Andre Araujo, William Schwartz
Foundation models have advanced computer vision by enabling strong performance across diverse tasks through large-scale pretraining and supervised fine-tuning. However, they may underperform in domains with distribution shifts and scarce labels, where supervised fine-tuning may be infeasible. While continued self-supervised learning for model adaptation is common for generative language models, this strategy has not proven effective for vision-centric encoder models. To address this challenge, we introduce a novel formulation of self-supervised fine-tuning for vision foundation models, where the model is adapted to a new domain without requiring new annotations. Our method, referred to as VESSA (Video-based Efficient SSL Adaptation), leverages two simple and effective insights. First, self-supervised fine-tuning is an important strategy that can be applied efficiently and simply, offering significant benefits. Second, to achieve meaningful improvements in the learning process, it is essential to use videos rather than still images during training. Through comprehensive experiments with 3 vision foundation models on 2 datasets, VESSA demonstrates consistent improvements in downstream tasks, compared to the base models and previous adaptation methods. Code and models will be released.