Robust SuperAlignment: Weak-to-Strong Robustness Generalization for Vision-Language Models
Junhao Dong, Cong Zhang, Xinghua Qu, Piotr Koniusz, Zejun MA, Yew Soon Ong
A spectrum of well-established studies has demonstrated the superhuman capabilities of modern Vision-Language Models (VLMs) across a wide range of tasks, inducing a growing doubt about continuing to obtain reliable supervision from human annotators. Therefore, concerns are growing about stagnating model performance due to scarce high-quality labelled data. To address this challenge, ``superalignment'' motivates the study of weak-to-strong generalization, where the supervision from a weak model can elicit generalizable knowledge from a strong one. While effective in aligning capabilities, we reveal that the standard weak-to-strong generalization scheme typically fails to confer robustness, exposing strong VLMs to underlying security threats brought by adversarial examples. Through systematic analysis, we attribute this failure mode to whether adversaries are incorporated into the learning objective during alignment. To address this, we propose the first adversarially robust weak-to-strong generalization method to elicit the zero-shot robustness from large-scale models in an unsupervised scheme by mitigating the unreliable supervision from two perspectives: alignment re-weighting and reference refining. We provide theoretical analyses characterizing the condition under which robustness generalization is achievable. Extensive experiments across various vision-language benchmarks validate the effectiveness and generalizability of our method, demonstrating its plug-and-play application to large-scale VLMs.