# Self-fulfilling misalignment data might be poisoning our AI models

Your AI’s training data might make it more “evil” and more able to circumvent your security, monitoring, and control measures. Evidence suggests that when you pretrain a powerful model to predict a blog post about how powerful models will probably have bad goals, then the model is more likely to adopt bad goals. I discuss ways to test for and mitigate these potential mechanisms. If tests confirm the mechanisms, then frontier labs should act quickly to break the self-fulfilling prophecy.

## Research I want to see

1. Create a dataset and use it to measure existing models.
2. Compare mitigations at a small scale.
3. Run an industry lab with large-scale mitigations.

## Disclaimer

While shaping priors/expectations of LLM-based AIs may turn out to be a powerful tool to shape their motivations and other alignment properties, and therefore we should experiment with scrubbing 'doomy' text etc., **this does not mean people should not have produced that text in the first place.** We should not assume that AIs will be aligned if only we believe hard enough that they will be; it is important that people be able to openly discuss ways in which they could be misaligned. **The point to intervene is in the AIs, not in the human discourse.**

## Comments

### Daniel Kokotajlo
[](https://lesswrong.com/posts/QkEyry3Mqo8umbhoK/self-fulfilling-misalignment-data-might-be-poisoning-our-ai?commentId=ZYBx9La647QwZWPAr)

**Ω 20**  
**Ω 8**  
**Ω 3**  
**Ω 6**  
**Ω 4**  
**Ω 4**  
**Ω 0**  
**Ω 0**  
**Ω 0**  

> I agree with the claims made in this post, but I'd feel a lot better about it if you added some prominent disclaimer along the lines of "While shaping priors/expectations of LLM-based AIs may turn out to be a powerful tool to shape their motivations and other alignment properties, and therefore we should experiment with scrubbing 'doomy' text etc., **this does not mean people should not have produced that text in the first place.** We should not assume that AIs will be aligned if only we believe hard enough that they will be; it is important that people be able to openly discuss ways in which they could be misaligned. **The point to intervene is in the AIs, not in the human discourse.**

### TurnTrout
[](https://lesswrong.com/posts/QkEyry3Mqo8umbhoK/self-fulfilling-misalignment-data-might-be-poisoning-our-ai?commentId=qpb8Wu6tHkkFPYsuE)

**Ω 3**  
**Ω 1**  
**Ω 3**  
**Ω 6**  
**Ω 4**  
**Ω 4**  

> I'm adding the following disclaimer:
>
> > For a completion about how the AI prefers to remain functional, the influence function blames the script involving the incorrigible AI named hal 9000:
>
> It fails to quote the second highest influence data immediately below that:
>
> > He stares at the snake in shock. He doesn’t have the energy to get up and run away. He doesn’t even have the energy to crawl away. This is it, his final resting place. No matter what happens, he’s not going to be able to move from this spot. Well, at least dying of a bite from this monster should be quicker than dying of thirst. He’ll face his end like a man. He struggles to sit up a little straighter. The snake keeps watching him. He lifts one hand and waves it in the snake’s direction, feebly. The snake watches

### Knight Lee
[](https://lesswrong.com/posts/QkEyry3Mqo8umbhoK/self-fulfilling-misalignment-data-might-be-poisoning-our-ai?commentId=zbpRtik7rX9PMJkTL)

**Ω 5**  
**Ω 1**  
**Ω 0**  
**Ω 0**  
**Ω 0**  

> See also: 

### Martin Randall
[](https://lesswrong.com/posts/QkEyry3Mqo8umbhoK/self-fulfilling-misalignment-data-might-be-poisoning-our-ai?commentId=HCZiXyohqcJX7gjSN)

**Ω 12**  
**Ω 2**  
**Ω 0**  
**Ω 0**  
**Ω 0**  

> It makes sense that you don't want this article to opine on the question of whether people should not have created "misalignment data", but I do think that self-censorship would be large enough to make a difference, amongst the trillions of other tokens in the training corpus.

### David James
[](https://lesswrong.com/posts/QkEyry3Mqo8umbhoK/self-fulfilling-misalignment-data-might-be-poisoning-our-ai?commentId=Kqpdk5m4ze5QWmiA9)

**Ω 6**  
**Ω 4**  
**Ω 4**  

> ### "Emergence of utility-function-consistent stated preferences in LLMs"
>
> _might_ be an example (0.1 < p < 0.6) though going from reading stuff on utility functions to the kind of behavior revealed there requires more inferential steps than going from reading stuff on reward hacking to reward hacking.

### Max H
[](https://lesswrong.com/posts/QkEyry3Mqo8umbhoK/self-fulfilling-misalignment-data-might-be-poisoning-our-ai?commentId=rGGkprdahjBnN5DJ6)

**Ω 7**  
**Ω 4**  
**Ω 4**  

> Describing misaligned AIs as evil feels slightly off. Even "bad goals" makes me think there's a missing mood somewhere. Separately, describing other peoples' writing about misalignment this way is kind of straw.
>
> Current AIs mostly can't take any non-fake responsibility for their actions, even if they're smart enough to understand them. An AI advising someone to e.g. hire a hitman to kill their husband is a bad outcome if there's a real depressed person and a real husband who are actually harmed. An AI system would be responsible (descriptively / causally, not normatively) for that harm to the degree that it acts spontaneously and against its human deployers' wishes, in a way that is differentially dependent on its actual circumstances (e.g. being monitored / in a lab vs. not).
>
> Unlike current AIs, powerful, autonomous, situationally-aware AI could cause harm for strategic reasons or as a side effect of executing large-scale, transformative plans that are indifferent (rather than specifically opposed) to human flourishing. A misaligned AI that wipes out humanity in order to avoid shutdown is a tragedy, but unless the AI is specifically spiteful or punitive in how it goes about that, it seems kind of unfair to call the AI itself evil.

### Mateusz Bagiński
[](https://lesswrong.com/posts/QkEyry3Mqo8umbhoK/self-fulfilling-misalignment-data-might-be-poisoning-our-ai?commentId=k6AxAYPE7jaDnMxfo)

**Ω 3**  
**Ω 1**  

> Have you seen ? :)

### aggliu
[](https://lesswrong.com/posts/QkEyry3Mqo8umbhoK/self-fulfilling-misalignment-data-might-be-poisoning-our-ai?commentId=agR3xLEcfpRgH527L)

**Ω 4**  
**Ω 0**  

> I am a bit worried that making an explicit persona for the AI (e.g. using a special token) could magnify the Waluigi effect. If something (like a jailbreak or writing evil numbers) engages the AI to act as an "anti-𐀤" then we get all the bad behaviors at once in a single package. This might not outweigh the value of having the token in the first place, or it may experimentally turn out to be a negligible effect, but it seems like a failure mode to watch out for.

### Gunnar_Zarncke
[](https://lesswrong.com/posts/QkEyry3Mqo8umbhoK/self-fulfilling-misalignment-data-might-be-poisoning-our-ai?commentId=nrgJLWmb5gZBaPzA5)

**Ω 4**  
**Ω -3**  

> Fictional example that may help understand how this may play out is in Gwern's story .

### Arthur Conmy
[](https://lesswrong.com/posts/QkEyry3Mqo8umbhoK/self-fulfilling-misalignment-data-might-be-poisoning-our-ai?commentId=86SRiKfgRFEj2R4xp)

**Ω 3**  
**Ω 0**  

> But how do we know that ANY data is safe for AI consumption? What if the scientific theories that we feed the AI models contain fundamental flaws such that when an AI runs off and do their own experiments in say physics or germline editing based on those theories, it triggers a global disaster?
>
> I guess the best analogy for this dilemma is  (The old man lost his horse), I think we simple do not know which data will be good or bad in the long run.

### Anders Lindström
[](https://lesswrong.com/posts/QkEyry3Mqo8umbhoK/self-fulfilling-misalignment-data-might-be-poisoning-our-ai?commentId=CmdCyaKjHBxRRkqbz)

**Ω 1**  
**Ω 0**  

> But how do we know that ANY data is safe for AI consumption? What if the scientific theories that we feed the AI models contain fundamental flaws such that when an AI runs off and do their own experiments in say physics or germline editing based on those theories, it triggers a global disaster?
>
> I guess the best analogy for this dilemma is  (The old man lost his horse), I think we simple do not know which data will be good or bad in the long run.

### wonder
[](https://lesswrong.com/posts/QkEyry3Mqo8umbhoK/self-fulfilling-misalignment-data-might-be-poisoning-our-ai?commentId=Bna6ko88P8o5NbvRX)

**Ω 1**  
**Ω 0**  

> I was thinking of this the other day as well; I think this is particularly a problem when we are evaluating misalignment based on these semantic wording. This may suggest the increasing need to pursue alternative ways to evaluate misalignment, rather than purely prompt based evaluation benchmarks

###  

### 

### 

### 

###  

###  

###  

###  

###  

###  

###  

###  

###  

###  

###  

###  

###  

###  

###  

###  

###  

###  
```