Cascading Adversarial Bias from Injection to Distillation in Language
  Models
Model distillation has become essential for creating smaller, deployable
language models that retain larger system capabilities. However, widespread
deployment raises concerns about resilience to adversarial manipulation. This
paper investigates vulnerability of distilled models to adversarial injection
of biased content during training. We demonstrate that adversaries can inject
subtle biases into teacher models through minimal data poisoning, which
propagates to student models and becomes significantly amplified. We propose
two propagation modes: Untargeted Propagation, where bias affects multiple
tasks, and Targeted Propagation, focusing on specific tasks while maintaining
normal behavior elsewhere. With only 25 poisoned samples (0.25% poisoning
rate), student models generate biased responses 76.9% of the time in targeted
scenarios - higher than 69.4% in teacher models. For untargeted propagation,
adversarial bias appears 6x-29x more frequently in student models on unseen
tasks. We validate findings across six bias types (targeted advertisements,
phishing links, narrative manipulations, insecure coding practices), various
distillation methods, and different modalities spanning text and code
generation. Our evaluation reveals shortcomings in current defenses -
perplexity filtering, bias detection systems, and LLM-based autorater
frameworks - against these attacks. Results expose significant security
vulnerabilities in distilled models, highlighting need for specialized
safeguards. We propose practical design principles for building effective
adversarial bias mitigation strategies.
683e9da357738c5cc36d5175
Harsh Chaudhari
683e9da357738c5cc36d5176
Jamie Hayes
683e9da357738c5cc36d5177
Matthew Jagielski
683e9da357738c5cc36d5178
Ilia Shumailov
683e9da357738c5cc36d5179
Milad Nasr
683e9da357738c5cc36d517a
Alina Oprea
Cascading Adversarial Bias from Injection to Distillation in Language
  Models
Model distillation has become essential for creating smaller, deployable
language models that retain larger system capabilities. However, widespread
deployment raises concerns about resilience to adversarial manipulation. This
paper investigates vulnerability of distilled models to adversarial injection
of biased content during training. We demonstrate that adversaries can inject
subtle biases into teacher models through minimal data poisoning, which
propagates to student models and becomes significantly amplified. We propose
two propagation modes: Untargeted Propagation, where bias affects multiple
tasks, and Targeted Propagation, focusing on specific tasks while maintaining
normal behavior elsewhere. With only 25 poisoned samples (0.25% poisoning
rate), student models generate biased responses 76.9% of the time in targeted
scenarios - higher than 69.4% in teacher models. For untargeted propagation,
adversarial bias appears 6x-29x more frequently in student models on unseen
tasks. We validate findings across six bias types (targeted advertisements,
phishing links, narrative manipulations, insecure coding practices), various
distillation methods, and different modalities spanning text and code
generation. Our evaluation reveals shortcomings in current defenses -
perplexity filtering, bias detection systems, and LLM-based autorater
frameworks - against these attacks. Results expose significant security
vulnerabilities in distilled models, highlighting need for specialized
safeguards. We propose practical design principles for building effective
adversarial bias mitigation strategies.
Adversarial injection of biased content can significantly propagate from teacher to student models during distillation, leading to frequent biased responses in both targeted and untargeted scenarios across various bias types and modalities.
model distillation language models adversarial manipulation data poisoning bias injection Untargeted Propagation Targeted Propagation perplexity filtering bias detection systems LLM-based autorater frameworks