Scaling Instructable Agents Across Many Simulated Worlds
Building embodied AI systems that can follow arbitrary language instructions
in any 3D environment is a key challenge for creating general AI. Accomplishing
this goal requires learning to ground language in perception and embodied
actions, in order to accomplish complex tasks. The Scalable, Instructable,
Multiworld Agent (SIMA) project tackles this by training agents to follow
free-form instructions across a diverse range of virtual 3D environments,
including curated research environments as well as open-ended, commercial video
games. Our goal is to develop an instructable agent that can accomplish
anything a human can do in any simulated 3D environment. Our approach focuses
on language-driven generality while imposing minimal assumptions. Our agents
interact with environments in real-time using a generic, human-like interface:
the inputs are image observations and language instructions and the outputs are
keyboard-and-mouse actions. This general approach is challenging, but it allows
agents to ground language across many visually complex and semantically rich
environments while also allowing us to readily run agents in new environments.
In this paper we describe our motivation and goal, the initial progress we have
made, and promising preliminary results on several diverse research
environments and a variety of commercial video games.
661fd19136201a18e5030fa8
SIMA Team
661fd19136201a18e5030fa9
Maria Abi Raad
661fd19136201a18e5030faa
Arun Ahuja
661fd19136201a18e5030fab
Catarina Barros
661fd19136201a18e5030fac
Frederic Besse
661fd19136201a18e5030fad
Andrew Bolt
661fd19136201a18e5030fae
Adrian Bolton
661fd19136201a18e5030faf
Bethanie Brownfield
661fd19136201a18e5030fb0
Gavin Buttimore
661fd19136201a18e5030fb1
Max Cant
661fd19136201a18e5030fb2
Sarah Chakera
661fd19136201a18e5030fb3
Stephanie C. Y. Chan
661fd19136201a18e5030fb4
Jeff Clune
661fd19136201a18e5030fb5
Adrian Collister
661fd19136201a18e5030fb6
Vikki Copeman
661fd19136201a18e5030fb7
Alex Cullum
661fd19136201a18e5030fb8
Ishita Dasgupta
admin_assigned
2024-04-18T08:27:20.958Z
64f7f2871435cc7faa39104d
/avatars/8d42adabf6610eae5169be3ac42165e3.svg
Ishita Dasgupta
user
ishitadg
661fd19136201a18e5030fb9
Dario de Cesare
661fd19136201a18e5030fba
Julia Di Trapani
661fd19136201a18e5030fbb
Yani Donchev
661fd19136201a18e5030fbc
Emma Dunleavy
661fd19136201a18e5030fbd
Martin Engelcke
661fd19136201a18e5030fbe
Ryan Faulkner
661fd19136201a18e5030fbf
Frankie Garcia
661fd19136201a18e5030fc0
Charles Gbadamosi
661fd19136201a18e5030fc1
Zhitao Gong
661fd19136201a18e5030fc2
Lucy Gonzales
661fd19136201a18e5030fc3
Karol Gregor
661fd19136201a18e5030fc4
Arne Olav Hallingstad
661fd19136201a18e5030fc5
Tim Harley
661fd19136201a18e5030fc6
Sam Haves
661fd19136201a18e5030fc7
Felix Hill
661fd19136201a18e5030fc8
Ed Hirst
claimed_verified
2024-08-16T14:45:50.085Z
66bf0fea094fe1e139af008a
/avatars/5b5a817bccb9d0f94f374426bbc6f775.svg
Ed Hirst
user
edhirst
661fd19136201a18e5030fc9
Drew A. Hudson
661fd19136201a18e5030fca
Steph Hughes-Fitt
661fd19136201a18e5030fcb
Danilo J. Rezende
661fd19136201a18e5030fcc
Mimi Jasarevic
661fd19136201a18e5030fcd
Laura Kampis
661fd19136201a18e5030fce
Rosemary Ke
661fd19136201a18e5030fcf
Thomas Keck
661fd19136201a18e5030fd0
Junkyung Kim
661fd19136201a18e5030fd1
Oscar Knagg
admin_assigned
2024-04-18T08:30:17.815Z
63204ed3f7db36538ca102e8
/avatars/c61db6e2ac29776eb552bda0b6adef58.svg
Oscar Knagg
user
oscarknagg
661fd19136201a18e5030fd2
Kavya Kopparapu
661fd19136201a18e5030fd3
Andrew Lampinen
661fd19136201a18e5030fd4
Shane Legg
661fd19136201a18e5030fd5
Alexander Lerchner
661fd19136201a18e5030fd6
Marjorie Limont
661fd19136201a18e5030fd7
Yulan Liu
661fd19136201a18e5030fd8
Maria Loks-Thompson
661fd19136201a18e5030fd9
Joseph Marino
661fd19136201a18e5030fda
Kathryn Martin Cussons
661fd19136201a18e5030fdb
Loic Matthey
661fd19136201a18e5030fdc
Siobhan Mcloughlin
661fd19136201a18e5030fdd
Piermaria Mendolicchio
661fd19136201a18e5030fde
Hamza Merzic
661fd19136201a18e5030fdf
Anna Mitenkova
661fd19136201a18e5030fe0
Alexandre Moufarek
661fd19136201a18e5030fe1
Valeria Oliveira
661fd19136201a18e5030fe2
Yanko Oliveira
661fd19136201a18e5030fe3
Hannah Openshaw
661fd19136201a18e5030fe4
Renke Pan
661fd19136201a18e5030fe5
Aneesh Pappu
661fd19136201a18e5030fe6
Alex Platonov
661fd19136201a18e5030fe7
Ollie Purkiss
661fd19136201a18e5030fe8
David Reichert
661fd19136201a18e5030fe9
John Reid
661fd19136201a18e5030fea
Pierre Harvey Richemond
661fd19136201a18e5030feb
Tyson Roberts
661fd19136201a18e5030fec
Giles Ruscoe
661fd19136201a18e5030fed
Jaume Sanchez Elias
661fd19136201a18e5030fee
Tasha Sandars
661fd19136201a18e5030fef
Daniel P. Sawyer
661fd19136201a18e5030ff0
Tim Scholtes
661fd19136201a18e5030ff1
Guy Simmons
661fd19136201a18e5030ff2
Daniel Slater
661fd19136201a18e5030ff3
Hubert Soyer
661fd19136201a18e5030ff4
Heiko Strathmann
661fd19136201a18e5030ff5
Peter Stys
661fd19136201a18e5030ff6
Allison C. Tam
661fd19136201a18e5030ff7
Denis Teplyashin
661fd19136201a18e5030ff8
Tayfun Terzi
661fd19136201a18e5030ff9
Davide Vercelli
661fd19136201a18e5030ffa
Bojan Vujatovic
661fd19136201a18e5030ffb
Marcus Wainwright
661fd19136201a18e5030ffc
Jane X. Wang
661fd19136201a18e5030ffd
Zhengdong Wang
661fd19136201a18e5030ffe
Daan Wierstra
661fd19136201a18e5030fff
Duncan Williams
661fd19136201a18e5031000
Nathaniel Wong
661fd19136201a18e5031001
Sarah York
661fd19136201a18e5031002
Nick Young
Scaling Instructable Agents Across Many Simulated Worlds
Building embodied AI systems that can follow arbitrary language instructions
in any 3D environment is a key challenge for creating general AI. Accomplishing
this goal requires learning to ground language in perception and embodied
actions, in order to accomplish complex tasks. The Scalable, Instructable,
Multiworld Agent (SIMA) project tackles this by training agents to follow
free-form instructions across a diverse range of virtual 3D environments,
including curated research environments as well as open-ended, commercial video
games. Our goal is to develop an instructable agent that can accomplish
anything a human can do in any simulated 3D environment. Our approach focuses
on language-driven generality while imposing minimal assumptions. Our agents
interact with environments in real-time using a generic, human-like interface:
the inputs are image observations and language instructions and the outputs are
keyboard-and-mouse actions. This general approach is challenging, but it allows
agents to ground language across many visually complex and semantically rich
environments while also allowing us to readily run agents in new environments.
In this paper we describe our motivation and goal, the initial progress we have
made, and promising preliminary results on several diverse research
environments and a variety of commercial video games.
Developing a scalable and adaptable embodied AI agent capable of following complex language instructions across various 3D environments using minimal assumptions.
embodied AI language instructions 3D environments generality image observations keyboard-and-mouse actions visual complexity semantic richness