Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant
  Safeguards into Open-Weight LLMs
Open-weight AI systems offer unique benefits, including enhanced
transparency, open research, and decentralized access. However, they are
vulnerable to tampering attacks which can efficiently elicit harmful behaviors
by modifying weights or activations. Currently, there is not yet a robust
science of open-weight model risk management. Existing safety fine-tuning
methods and other post-training techniques have struggled to make LLMs
resistant to more than a few dozen steps of adversarial fine-tuning. In this
paper, we investigate whether filtering text about dual-use topics from
training data can prevent unwanted capabilities and serve as a more
tamper-resistant safeguard. We introduce a multi-stage pipeline for scalable
data filtering and show that it offers a tractable and effective method for
minimizing biothreat proxy knowledge in LLMs. We pretrain multiple
6.9B-parameter models from scratch and find that they exhibit substantial
resistance to adversarial fine-tuning attacks on up to 10,000 steps and 300M
tokens of biothreat-related text -- outperforming existing post-training
baselines by over an order of magnitude -- with no observed degradation to
unrelated capabilities. However, while filtered models lack internalized
dangerous knowledge, we find that they can still leverage such information when
it is provided in context (e.g., via search tool augmentation), demonstrating a
need for a defense-in-depth approach. Overall, these findings help to establish
pretraining data curation as a promising layer of defense for open-weight AI
systems.
689aa13ffab6fdd2e52ac416
Kyle O'Brien
claimed_verified
2025-08-13T07:21:16.285Z
6292cbb184bbf63394ed3c5e
https://cdn-avatars.huggingface.co/v1/production/uploads/6292cbb184bbf63394ed3c5e/rramXFaiKPoLUkD25XF1u.jpeg
Kyle O'Brien
user
Kyle1668
689aa13ffab6fdd2e52ac417
Stephen Casper
689aa13ffab6fdd2e52ac418
Quentin Anthony
689aa13ffab6fdd2e52ac419
Tomek Korbak
689aa13ffab6fdd2e52ac41a
Robert Kirk
689aa13ffab6fdd2e52ac41b
Xander Davies
689aa13ffab6fdd2e52ac41c
Ishan Mishra
689aa13ffab6fdd2e52ac41d
Geoffrey Irving
689aa13ffab6fdd2e52ac41e
Yarin Gal
689aa13ffab6fdd2e52ac41f
Stella Biderman
Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant
  Safeguards into Open-Weight LLMs
Open-weight AI systems offer unique benefits, including enhanced
transparency, open research, and decentralized access. However, they are
vulnerable to tampering attacks which can efficiently elicit harmful behaviors
by modifying weights or activations. Currently, there is not yet a robust
science of open-weight model risk management. Existing safety fine-tuning
methods and other post-training techniques have struggled to make LLMs
resistant to more than a few dozen steps of adversarial fine-tuning. In this
paper, we investigate whether filtering text about dual-use topics from
training data can prevent unwanted capabilities and serve as a more
tamper-resistant safeguard. We introduce a multi-stage pipeline for scalable
data filtering and show that it offers a tractable and effective method for
minimizing biothreat proxy knowledge in LLMs. We pretrain multiple
6.9B-parameter models from scratch and find that they exhibit substantial
resistance to adversarial fine-tuning attacks on up to 10,000 steps and 300M
tokens of biothreat-related text -- outperforming existing post-training
baselines by over an order of magnitude -- with no observed degradation to
unrelated capabilities. However, while filtered models lack internalized
dangerous knowledge, we find that they can still leverage such information when
it is provided in context (e.g., via search tool augmentation), demonstrating a
need for a defense-in-depth approach. Overall, these findings help to establish
pretraining data curation as a promising layer of defense for open-weight AI
systems.
Data filtering during pretraining enhances LLM resistance to adversarial fine-tuning attacks without degrading unrelated capabilities, offering a promising defense mechanism for open-weight AI systems.
open-weight AI systems tampering attacks adversarial fine-tuning data filtering multi-stage pipeline biothreat proxy knowledge pretraining defense-in-depth