# Will LLM agents become the first takeover-capable AGIs?

## Question
**by Seth Herd**


**36** votes

### 1. 36

### 2. 36

### 3. 36

### 4. 36

### 5. 14

### 6. 36

### 7. 8

### 8. 12

### 9. 9

### 10. 36

## Answer
One of my takeaways from EA Global this year was that most alignment people aren't explicitly focused on LLM-based agents (LMAs) as a route to takeover-capable AGI. I want to better understand this position, since I estimate this path to AGI as likely enough (maybe around 60%) to be worth specific focus and concern.

Two reasons people might not care about aligning LMAs in particular:
1. Thinking this route to AGI is quite possible but that aligning LLMs mostly covers aligning LLM agents
2. Thinking LLM-based agents are unlikely to be the first takeover-capable AGI

I'm aware of arguments/questions like  and , and LLMs' weakness on what Steve Byrnes : the ability to tell their better ideas/outputs from their worse ones. I'm curious if these or other ideas play a major role in your thinking.

I'm even more curious about the distribution of opinions around type 1 (aligning LLMs covers aligning LMAs) and 2 (LMAs are not a likely route to AGI) in the alignment community.

Edit: Based on the comments, I think perhaps this question is too broadly stated. The better question is "what sort of LMAs do you expect to reach takeover-capable AGIs?"

### 1. 1
For these purposes I want to consider language model agents (LMAs) broadly. I mean any sort of system that uses models that are substantially trained on human language, similar to current GPTs trained primarily to predict human language use.

Agents based on language models could be systems with a lot or a little scaffolding (including but not limited to hard-coded prompts for different cognitive purposes), and other cognitive systems (including but not limited to dedicated one-shot memory systems and executive function/planning or metacognition systems). This is a large category of models, but they have important similarities for alignment purposes: LLMs generate their "thoughts", while other systems direct and modify those "thoughts", to both organizing and chaotic effect.

This of course includes multimodal foundation models that include natural language training as a major component; most current things we call LLMs are technically foundation models. I think language training is the most important bit. I suspect that language training is remarkably effective because human language is a high-effort distillation of the world's semantics; but that is another story.

### 2. 2
I think that humans are also relatively weak at generating novel insights, generalizing, and discernment using our System 1 processing. I think that agentic scaffolding and training is likely to improve System 2 strategies and skills similar to those humans use to scrape by in those areas.

### 3. 3
 is my brief abstract argument for why there are no breakthroughs needed for this route to AGI,  summarizes the plan for aligning them in short timelines; and  is my latest in-depth prediction on how labs will try to align them by default, and how those methods could succeed or fail.

### 4. 4
### 5. 2
### 6. 2

## Comments
### 1. 1
**Vladimir_Nesov**


### 2. 2
**Seth Herd**


### 3. 3
**CBiddulph**


### 4. 4
**Seth Herd**


### 5. 5
**Campbell Hutcheson**


### 6. 6
**Seth Herd**


### 7. 7
**Indigo3**


### 8. 8
**tailcalled**


### 9. 9
**Seth Herd**


### 10. 10
**Seth Herd**


## Moderation Log

```