Reconstructing Robust Vision-Language Models from Natural Latent Spaces
Zhangyun Wang, Ni Ding, Aniket Mahanti
Pre-trained Vision-Language Models (VLMs) exhibit significant vulnerability to imperceptible adversarial perturbations. Current advanced defense strategies typically employ adversarial prompt tuning to improve the adversarial robustness of VLMs, which struggle to simultaneously maintain generalization across both natural and adversarial examples under different benchmarks and downstream tasks. We propose a Collaborative Adversarial Prompt Tuning (CoAPT) approach from pre-trained models to target robust models. Specifically, we adopt an improved fast total variation (TV) technique to suppress and eliminate high-frequency details from images while preserving edge structures, thereby disrupting the adversarial perturbation space. Subsequently, guided by the high-level image and text representations in the latent space of the pre-trained VLMs, the corrupted natural features are restored while inheriting the superior generalization capability. Compared to existing state-of-the-art methods, CoAPT achieves a superior trade-off between robustness and generalization. Across four benchmark tasks, CoAPT improves robustness by 28.37\% (few-shot-16), 31.67\% (base-to-novel), 20.01\% (zero-shot), and 17.37\% (out-of-distribution), while maintaining an average increase of 9.83\% in natural accuracy.