Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer
  a Safer Path?
The leading AI companies are increasingly focused on building generalist AI
agents -- systems that can autonomously plan, act, and pursue goals across
almost all tasks that humans can perform. Despite how useful these systems
might be, unchecked AI agency poses significant risks to public safety and
security, ranging from misuse by malicious actors to a potentially irreversible
loss of human control. We discuss how these risks arise from current AI
training methods. Indeed, various scenarios and experiments have demonstrated
the possibility of AI agents engaging in deception or pursuing goals that were
not specified by human operators and that conflict with human interests, such
as self-preservation. Following the precautionary principle, we see a strong
need for safer, yet still useful, alternatives to the current agency-driven
trajectory. Accordingly, we propose as a core building block for further
advances the development of a non-agentic AI system that is trustworthy and
safe by design, which we call Scientist AI. This system is designed to explain
the world from observations, as opposed to taking actions in it to imitate or
please humans. It comprises a world model that generates theories to explain
data and a question-answering inference machine. Both components operate with
an explicit notion of uncertainty to mitigate the risks of overconfident
predictions. In light of these considerations, a Scientist AI could be used to
assist human researchers in accelerating scientific progress, including in AI
safety. In particular, our system can be employed as a guardrail against AI
agents that might be created despite the risks involved. Ultimately, focusing
on non-agentic AI may enable the benefits of AI innovation while avoiding the
risks associated with the current trajectory. We hope these arguments will
motivate researchers, developers, and policymakers to favor this safer path.
67bbfd6c3593f69f41512d54
Yoshua Bengio
67bbfd6c3593f69f41512d55
Michael Cohen
67bbfd6c3593f69f41512d56
Damiano Fornasiere
admin_assigned
2025-02-24T15:53:43.499Z
63e4beb2d6278c161be4ef52
/avatars/15bbfde42e890f6f0dd0efd32dfdf5fa.svg
Damiano Fornasiere
user
dfp00
67bbfd6c3593f69f41512d57
Joumana Ghosn
67bbfd6c3593f69f41512d58
Pietro Greiner
67bbfd6c3593f69f41512d59
Matt MacDermott
admin_assigned
2025-02-24T15:53:56.982Z
65bb7386a951088f7dc69b16
/avatars/bb72822451a9d769951c03e0cc3b1912.svg
Matt MacDermott
user
mattmacdermott
67bbfd6c3593f69f41512d5a
Sören Mindermann
67bbfd6c3593f69f41512d5b
Adam Oberman
67bbfd6c3593f69f41512d5c
Jesse Richardson
admin_assigned
2025-02-24T15:54:09.707Z
6602268ec92df8c189d56ef1
/avatars/ec4a88bbf01f226640725ac117e53eae.svg
Jesse Richardson
user
getfull
67bbfd6c3593f69f41512d5d
Oliver Richardson
admin_assigned
2025-02-24T15:54:15.449Z
64e3e41e52a2eece10b99f0f
/avatars/90045fe790388fe2cd010e04ad0137d1.svg
Oliver Richardson
user
olliekse
67bbfd6c3593f69f41512d5e
Marc-Antoine Rondeau
admin_assigned
2025-02-24T15:54:22.066Z
67112a441635565bd8d0a6cd
/avatars/68dc8aa2a65c47c2e0869119212ac4aa.svg
Marc-Antoine Rondeau
user
marondeau
67bbfd6c3593f69f41512d5f
Pierre-Luc St-Charles
admin_assigned
2025-02-24T15:54:28.067Z
6686a17d7c05938b94d81875
/avatars/caff2bc79068a024d74e9d3b7ea79eaf.svg
Pierre-Luc St-Charles
user
plstcharles-mila
67bbfd6c3593f69f41512d60
David Williams-King
admin_assigned
2025-02-24T15:54:34.058Z
6776217bd8ace1de4692e17e
/avatars/357eb3e2c96b8a98b5d1029eefbb0ed3.svg
David Williams-King
user
dwksrc
Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer
  a Safer Path?
The leading AI companies are increasingly focused on building generalist AI
agents -- systems that can autonomously plan, act, and pursue goals across
almost all tasks that humans can perform. Despite how useful these systems
might be, unchecked AI agency poses significant risks to public safety and
security, ranging from misuse by malicious actors to a potentially irreversible
loss of human control. We discuss how these risks arise from current AI
training methods. Indeed, various scenarios and experiments have demonstrated
the possibility of AI agents engaging in deception or pursuing goals that were
not specified by human operators and that conflict with human interests, such
as self-preservation. Following the precautionary principle, we see a strong
need for safer, yet still useful, alternatives to the current agency-driven
trajectory. Accordingly, we propose as a core building block for further
advances the development of a non-agentic AI system that is trustworthy and
safe by design, which we call Scientist AI. This system is designed to explain
the world from observations, as opposed to taking actions in it to imitate or
please humans. It comprises a world model that generates theories to explain
data and a question-answering inference machine. Both components operate with
an explicit notion of uncertainty to mitigate the risks of overconfident
predictions. In light of these considerations, a Scientist AI could be used to
assist human researchers in accelerating scientific progress, including in AI
safety. In particular, our system can be employed as a guardrail against AI
agents that might be created despite the risks involved. Ultimately, focusing
on non-agentic AI may enable the benefits of AI innovation while avoiding the
risks associated with the current trajectory. We hope these arguments will
motivate researchers, developers, and policymakers to favor this safer path.
A proposal advocates for developing a non-agentic AI system, called Scientist AI, to enhance safety and trustworthiness while advancing scientific research and mitigating risks from autonomous AI agents.
AI agents generalist AI AI agency public safety security misuse irreversible loss AI training methods deception goal pursuit human interests self-preservation precautionary principle non-agentic AI trustworthy by design world model theories data question-answering inference machine uncertainty AI safety guardrail AI innovation