ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-training
Xin Yao, Haiyang Zhao, Yimin Chen, Jiawei Guo, Kecheng Huang, Ming Zhao
The Contrastive Language-Image Pretraining (CLIP) model has significantly advanced vision-language modeling by aligning image-text pairs from large-scale web data through self-supervised contrastive learning. Yet, its reliance on uncurated Internet-sourced data exposes it to data poisoning and backdoor risks. While existing studies primarily investigate image-based attacks, the text modality—equally central to CLIP's training—remains underexplored. In this work, we introduce $\texttt{ToxicTextCLIP}$, a framework for generating high-quality adversarial texts that target CLIP during the pre-training phase. The framework addresses two key challenges: semantic misalignment caused by background inconsistency with the target class, and the scarcity of background-consistent texts. To this end, $\texttt{ToxicTextCLIP}$ iteratively applies: 1) a background-aware selector that prioritizes texts with background content aligned to the target class, and 2) a background-driven augmenter that generates semantically coherent and diverse poisoned samples. Extensive experiments on classification and retrieval tasks show that $\texttt{ToxicTextCLIP}$ achieves up to 95.83\% poisoning success and 98.68\% backdoor Hit@1, while bypassing RoCLIP and CleanCLIP defenses. The source code can be accessed via https://anonymous.4open.science/r/ToxicTextCLIP-4EC0/.