AutoRT: Embodied Foundation Models for Large Scale Orchestration of
  Robotic Agents
Foundation models that incorporate language, vision, and more recently
actions have revolutionized the ability to harness internet scale data to
reason about useful tasks. However, one of the key challenges of training
embodied foundation models is the lack of data grounded in the physical world.
In this paper, we propose AutoRT, a system that leverages existing foundation
models to scale up the deployment of operational robots in completely unseen
scenarios with minimal human supervision. AutoRT leverages vision-language
models (VLMs) for scene understanding and grounding, and further uses large
language models (LLMs) for proposing diverse and novel instructions to be
performed by a fleet of robots. Guiding data collection by tapping into the
knowledge of foundation models enables AutoRT to effectively reason about
autonomy tradeoffs and safety while significantly scaling up data collection
for robot learning. We demonstrate AutoRT proposing instructions to over 20
robots across multiple buildings and collecting 77k real robot episodes via
both teleoperation and autonomous robot policies. We experimentally show that
such "in-the-wild" data collected by AutoRT is significantly more diverse, and
that AutoRT's use of LLMs allows for instruction following data collection
robots that can align to human preferences.
65b0828cc11502dcf18b86f6
Michael Ahn
65b0828cc11502dcf18b86f7
Debidatta Dwibedi
65b0828cc11502dcf18b86f8
Chelsea Finn
admin_assigned
2024-01-24T10:05:34.239Z
64ac22a9193f0a807deb673d
/avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
Chelsea Finn
user
cbfinn
65b0828cc11502dcf18b86f9
Montse Gonzalez Arenas
65b0828cc11502dcf18b86fa
Keerthana Gopalakrishnan
admin_assigned
2024-01-24T10:05:51.318Z
632f7dee34c562cb1d380fdc
/avatars/d689e260535b8aa775654160028d4ec8.svg
Keerthana Gopalakrishnan
user
keerthanpg
65b0828cc11502dcf18b86fb
Karol Hausman
admin_assigned
2024-01-24T10:05:59.131Z
64c93e623a3f428da67d9310
/avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
Karol Hausman
user
KarolHausman
65b0828cc11502dcf18b86fc
Brian Ichter
admin_assigned
2024-01-24T10:06:08.348Z
633a26ab474cfeb1a864dc56
/avatars/cd40accd894fc78810fc0d5108f413e9.svg
Brian I
user
brianichter
65b0828cc11502dcf18b86fd
Alex Irpan
65b0828cc11502dcf18b86fe
Nikhil Joshi
admin_assigned
2024-01-24T10:06:47.545Z
63c15485894342c8964c481d
/avatars/cd938a12eb73a8395872f4058d85b0c5.svg
nikhil joshi
user
valeska
65b0828cc11502dcf18b86ff
Ryan Julian
65b0828cc11502dcf18b8700
Sean Kirmani
admin_assigned
2024-01-24T10:06:59.532Z
648c85f33ffc11989bddcce1
/avatars/bdde73b95a36e32cb2975656cca46022.svg
Sean Kirmani
user
skirmani
65b0828cc11502dcf18b8701
Isabel Leal
65b0828cc11502dcf18b8702
Edward Lee
65b0828cc11502dcf18b8703
Sergey Levine
65b0828cc11502dcf18b8704
Yao Lu
65b0828cc11502dcf18b8705
Isabel Leal
65b0828cc11502dcf18b8706
Sharath Maddineni
65b0828cc11502dcf18b8707
Kanishka Rao
65b0828cc11502dcf18b8708
Dorsa Sadigh
65b0828cc11502dcf18b8709
Pannag Sanketi
claimed_verified
2024-08-09T07:49:17.408Z
66b56f73fc66a466e6104bfb
/avatars/74855aaf580144fafead4b93d6a24a46.svg
Pannag Sanketi
user
prsanky
65b0828cc11502dcf18b870a
Pierre Sermanet
65b0828cc11502dcf18b870b
Quan Vuong
admin_assigned
2024-01-24T10:08:51.376Z
63e1d6687fbb6ae4d4f4d980
/avatars/4959319e69d69e03a31ff4be326e602a.svg
Quan Vuong
user
quintonx
65b0828cc11502dcf18b870c
Stefan Welker
65b0828cc11502dcf18b870d
Fei Xia
admin_assigned
2024-01-24T10:11:23.433Z
62cfa9ee9b96f22525c36d47
/avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
Fei Xia
user
fxia22
65b0828cc11502dcf18b870e
Ted Xiao
65b0828cc11502dcf18b870f
Peng Xu
65b0828cc11502dcf18b8710
Steve Xu
65b0828cc11502dcf18b8711
Zhuo Xu
AutoRT: Embodied Foundation Models for Large Scale Orchestration of
  Robotic Agents
Foundation models that incorporate language, vision, and more recently
actions have revolutionized the ability to harness internet scale data to
reason about useful tasks. However, one of the key challenges of training
embodied foundation models is the lack of data grounded in the physical world.
In this paper, we propose AutoRT, a system that leverages existing foundation
models to scale up the deployment of operational robots in completely unseen
scenarios with minimal human supervision. AutoRT leverages vision-language
models (VLMs) for scene understanding and grounding, and further uses large
language models (LLMs) for proposing diverse and novel instructions to be
performed by a fleet of robots. Guiding data collection by tapping into the
knowledge of foundation models enables AutoRT to effectively reason about
autonomy tradeoffs and safety while significantly scaling up data collection
for robot learning. We demonstrate AutoRT proposing instructions to over 20
robots across multiple buildings and collecting 77k real robot episodes via
both teleoperation and autonomous robot policies. We experimentally show that
such "in-the-wild" data collected by AutoRT is significantly more diverse, and
that AutoRT's use of LLMs allows for instruction following data collection
robots that can align to human preferences.
AutoRT uses vision-language and large language models to enable operational robots to perform diverse tasks in unseen scenarios with minimal human supervision, enhancing data collection and alignment with human preferences.
vision-language models large language models embodied foundation models scene understanding guiding data collection autonomy tradeoffs safety instruction following data collection