VPA: Fully Test-Time Visual Prompt Adaptation
Textual prompt tuning has demonstrated significant performance improvements
in adapting natural language processing models to a variety of downstream tasks
by treating hand-engineered prompts as trainable parameters. Inspired by the
success of textual prompting, several studies have investigated the efficacy of
visual prompt tuning. In this work, we present Visual Prompt Adaptation (VPA),
the first framework that generalizes visual prompting with test-time
adaptation. VPA introduces a small number of learnable tokens, enabling fully
test-time and storage-efficient adaptation without necessitating source-domain
information. We examine our VPA design under diverse adaptation settings,
encompassing single-image, batched-image, and pseudo-label adaptation. We
evaluate VPA on multiple tasks, including out-of-distribution (OOD)
generalization, corruption robustness, and domain adaptation. Experimental
results reveal that VPA effectively enhances OOD generalization by 3.3% across
various models, surpassing previous test-time approaches. Furthermore, we show
that VPA improves corruption robustness by 6.5% compared to strong baselines.
Finally, we demonstrate that VPA also boosts domain adaptation performance by
relatively 5.2%. Our VPA also exhibits marked effectiveness in improving the
robustness of zero-shot recognition for vision-language models.
6514f5f30885410769abebe2
Jiachen Sun
extracted_pending
2023-09-28T03:41:40.652Z
643ed00b0d1194da249b7850
https://cdn-avatars.huggingface.co/v1/production/uploads/643ed00b0d1194da249b7850/AyrfPjxLM7BrHlAZzhsA0.jpeg
Jason
user
jcjcjcjcsun
6514f5f30885410769abebe3
Mark Ibrahim
extracted_confirmed
2024-11-18T15:18:15.898Z
64c17345e82e55936cf971bc
/avatars/b39c5637e00291410620afb7c770d587.svg
Mark Ibrahim
user
marksibrahim
6514f5f30885410769abebe4
Melissa Hall
extracted_confirmed
2024-07-17T13:00:53.313Z
6697bfb4acbced03ebd133f8
/avatars/a7d41367c6f7974303feb5af2a387ce4.svg
Melissa Hall
user
melissah14
6514f5f30885410769abebe5
Ivan Evtimov
extracted_pending
2024-06-27T23:59:15.505Z
667dbd8fcef5b7fa35acd25d
/avatars/d484c9034d79b465744c831b30f04623.svg
Ivan Evtimov
user
ivanevtimov
6514f5f30885410769abebe6
Z. Morley Mao
6514f5f30885410769abebe7
Cristian Canton Ferrer
admin_assigned
2023-09-28T09:20:47.831Z
63127ad9fa3e7ba13f17b11a
/avatars/3b91b48702488526511ac86fde4f5e79.svg
cristian ferrer
user
chiky222
6514f5f30885410769abebe8
Caner Hazirbas
claimed_verified
2023-09-28T19:03:40.682Z
6515b9414eb20107d01eba58
https://cdn-avatars.huggingface.co/v1/production/uploads/6515b9414eb20107d01eba58/8LSqNIgItKUVkWlEQERMD.jpeg
Caner Hazirbas
user
hazirbas
VPA: Fully Test-Time Visual Prompt Adaptation
Textual prompt tuning has demonstrated significant performance improvements
in adapting natural language processing models to a variety of downstream tasks
by treating hand-engineered prompts as trainable parameters. Inspired by the
success of textual prompting, several studies have investigated the efficacy of
visual prompt tuning. In this work, we present Visual Prompt Adaptation (VPA),
the first framework that generalizes visual prompting with test-time
adaptation. VPA introduces a small number of learnable tokens, enabling fully
test-time and storage-efficient adaptation without necessitating source-domain
information. We examine our VPA design under diverse adaptation settings,
encompassing single-image, batched-image, and pseudo-label adaptation. We
evaluate VPA on multiple tasks, including out-of-distribution (OOD)
generalization, corruption robustness, and domain adaptation. Experimental
results reveal that VPA effectively enhances OOD generalization by 3.3% across
various models, surpassing previous test-time approaches. Furthermore, we show
that VPA improves corruption robustness by 6.5% compared to strong baselines.
Finally, we demonstrate that VPA also boosts domain adaptation performance by
relatively 5.2%. Our VPA also exhibits marked effectiveness in improving the
robustness of zero-shot recognition for vision-language models.
Visual Prompt Adaptation (VPA) enhances the performance of natural language processing models on various tasks through test-time adaptation using learnable tokens, improving OOD generalization, corruption robustness, and domain adaptation.
Visual Prompt Adaptation VPA learnable tokens test-time adaptation storage-efficient adaptation single-image batched-image pseudo-label adaptation out-of-distribution OOD generalization corruption robustness domain adaptation zero-shot recognition vision-language models