Safeguarding Vision-Language Models: Mitigating Vulnerabilities to
  Gaussian Noise in Perturbation-based Attacks
Vision-Language Models (VLMs) extend the capabilities of Large Language
Models (LLMs) by incorporating visual information, yet they remain vulnerable
to jailbreak attacks, especially when processing noisy or corrupted images.
Although existing VLMs adopt security measures during training to mitigate such
attacks, vulnerabilities associated with noise-augmented visual inputs are
overlooked. In this work, we identify that missing noise-augmented training
causes critical security gaps: many VLMs are susceptible to even simple
perturbations such as Gaussian noise. To address this challenge, we propose
Robust-VLGuard, a multimodal safety dataset with aligned / misaligned
image-text pairs, combined with noise-augmented fine-tuning that reduces attack
success rates while preserving functionality of VLM. For stronger
optimization-based visual perturbation attacks, we propose DiffPure-VLM,
leveraging diffusion models to convert adversarial perturbations into
Gaussian-like noise, which can be defended by VLMs with noise-augmented safety
fine-tuning. Experimental results demonstrate that the distribution-shifting
property of diffusion model aligns well with our fine-tuned VLMs, significantly
mitigating adversarial perturbations across varying intensities. The dataset
and code are available at https://github.com/JarvisUSTC/DiffPure-RobustVLM.
67ede544ed9c94861b82b29f
Jiawei Wang
claimed_verified
2025-04-03T07:52:27.815Z
64060b49a577649430bf6974
/avatars/74d0d6ed656b593e4c101b09edf18c7a.svg
Jiawei Wang
user
Jarvis1111
67ede544ed9c94861b82b2a0
Yushen Zuo
admin_assigned
2025-04-03T14:04:22.576Z
643e9efa2263cdc630f88f5c
/avatars/96cea51f17e7d41ffb6a4b438e05f5cb.svg
Yushen Zuo
user
YSZuo
67ede544ed9c94861b82b2a1
Yuanjun Chai
claimed_verified
2025-04-03T07:52:23.962Z
64756323d815855e4ef945a0
/avatars/29f5150805dafce2b3f9da441c8be988.svg
Chai
user
AllenChai
67ede544ed9c94861b82b2a2
Zhendong Liu
admin_assigned
2025-04-03T14:04:31.772Z
6267a1aff91d1c1633bf3fc0
/avatars/c3a6be3e4cea8c447ea2ba18059a566c.svg
ZhenDong Liu
user
Zaneec
67ede544ed9c94861b82b2a3
Yichen Fu
extracted_pending
2025-04-03T01:32:57.886Z
6528ce81598467feb33d992d
/avatars/e98a7bf16e6fd5118e861d562f93bb9b.svg
Yicheng Fu
user
sofyc
67ede544ed9c94861b82b2a4
Yichun Feng
admin_assigned
2025-04-03T14:04:42.397Z
67b6e2769fd5603fe15b84b9
/avatars/e6b42c49954a840518bc3e3aa2617912.svg
fengyichun
user
fengyichun
67ede544ed9c94861b82b2a5
Kin-man Lam
Safeguarding Vision-Language Models: Mitigating Vulnerabilities to
  Gaussian Noise in Perturbation-based Attacks
Vision-Language Models (VLMs) extend the capabilities of Large Language
Models (LLMs) by incorporating visual information, yet they remain vulnerable
to jailbreak attacks, especially when processing noisy or corrupted images.
Although existing VLMs adopt security measures during training to mitigate such
attacks, vulnerabilities associated with noise-augmented visual inputs are
overlooked. In this work, we identify that missing noise-augmented training
causes critical security gaps: many VLMs are susceptible to even simple
perturbations such as Gaussian noise. To address this challenge, we propose
Robust-VLGuard, a multimodal safety dataset with aligned / misaligned
image-text pairs, combined with noise-augmented fine-tuning that reduces attack
success rates while preserving functionality of VLM. For stronger
optimization-based visual perturbation attacks, we propose DiffPure-VLM,
leveraging diffusion models to convert adversarial perturbations into
Gaussian-like noise, which can be defended by VLMs with noise-augmented safety
fine-tuning. Experimental results demonstrate that the distribution-shifting
property of diffusion model aligns well with our fine-tuned VLMs, significantly
mitigating adversarial perturbations across varying intensities. The dataset
and code are available at https://github.com/JarvisUSTC/DiffPure-RobustVLM.
Robust-VLGuard and DiffPure-VLM enhance Vision-Language Models by mitigating vulnerabilities to adversarial perturbations through noise-augmented training and diffusion models.
Vision-Language Models Large Language Models noise-augmented training adversarial perturbations Gaussian noise multimodal safety dataset noise-augmented fine-tuning diffusion models distribution-shifting property DiffPure-VLM Robust-VLGuard