Singular Value Few-shot Adaptation of Vision-Language Models
Vision-language models (VLMs) like CLIP have shown impressive zero-shot and
few-shot learning capabilities across diverse applications. However, adapting
these models to new fine-grained domains remains difficult due to reliance on
prompt engineering and the high cost of full model fine-tuning. Existing
adaptation approaches rely on augmented components, such as prompt tokens and
adapter modules, which could limit adaptation quality, destabilize the model,
and compromise the rich knowledge learned during pretraining. In this work, we
present CLIP-SVD, a novel multi-modal and
parameter-efficient adaptation technique that leverages Singular Value
Decomposition (SVD) to modify the internal parameter space of CLIP without
injecting additional modules. Specifically, we fine-tune only the singular
values of the CLIP parameter matrices to rescale the basis vectors for domain
adaptation while retaining the pretrained model. This design enables enhanced
adaptation performance using only 0.04\% of the model's total
parameters and better preservation of its generalization ability. CLIP-SVD
achieves state-of-the-art classification results on 11 natural and 10
biomedical datasets, outperforming previous methods in both accuracy and
generalization under few-shot settings. Additionally, we leverage a natural
language-based approach to analyze the effectiveness and dynamics of the CLIP
adaptation to allow interpretability of CLIP-SVD. The code is publicly
available at https://github.com/HealthX-Lab/CLIP-SVD.
68c0423d56a3252309f75cb9
Taha Koleilat
claimed_verified
2025-09-12T16:19:28.409Z
67d33a3b1d5d1fd92c3cfce4
https://cdn-avatars.huggingface.co/v1/production/uploads/67d33a3b1d5d1fd92c3cfce4/MFBB29WXxNz5vgwPY_GkK.png
Taha Koleilat
user
TahaKoleilat
68c0423d56a3252309f75cba
Hassan Rivaz
68c0423d56a3252309f75cbb
Yiming Xiao
Singular Value Few-shot Adaptation of Vision-Language Models
Vision-language models (VLMs) like CLIP have shown impressive zero-shot and
few-shot learning capabilities across diverse applications. However, adapting
these models to new fine-grained domains remains difficult due to reliance on
prompt engineering and the high cost of full model fine-tuning. Existing
adaptation approaches rely on augmented components, such as prompt tokens and
adapter modules, which could limit adaptation quality, destabilize the model,
and compromise the rich knowledge learned during pretraining. In this work, we
present CLIP-SVD, a novel multi-modal and
parameter-efficient adaptation technique that leverages Singular Value
Decomposition (SVD) to modify the internal parameter space of CLIP without
injecting additional modules. Specifically, we fine-tune only the singular
values of the CLIP parameter matrices to rescale the basis vectors for domain
adaptation while retaining the pretrained model. This design enables enhanced
adaptation performance using only 0.04\% of the model's total
parameters and better preservation of its generalization ability. CLIP-SVD
achieves state-of-the-art classification results on 11 natural and 10
biomedical datasets, outperforming previous methods in both accuracy and
generalization under few-shot settings. Additionally, we leverage a natural
language-based approach to analyze the effectiveness and dynamics of the CLIP
adaptation to allow interpretability of CLIP-SVD. The code is publicly
available at https://github.com/HealthX-Lab/CLIP-SVD.