Octo: An Open-Source Generalist Robot Policy
Large policies pretrained on diverse robot datasets have the potential to
transform robotic learning: instead of training new policies from scratch, such
generalist robot policies may be finetuned with only a little in-domain data,
yet generalize broadly. However, to be widely applicable across a range of
robotic learning scenarios, environments, and tasks, such policies need to
handle diverse sensors and action spaces, accommodate a variety of commonly
used robotic platforms, and finetune readily and efficiently to new domains. In
this work, we aim to lay the groundwork for developing open-source, widely
applicable, generalist policies for robotic manipulation. As a first step, we
introduce Octo, a large transformer-based policy trained on 800k trajectories
from the Open X-Embodiment dataset, the largest robot manipulation dataset to
date. It can be instructed via language commands or goal images and can be
effectively finetuned to robot setups with new sensory inputs and action spaces
within a few hours on standard consumer GPUs. In experiments across 9 robotic
platforms, we demonstrate that Octo serves as a versatile policy initialization
that can be effectively finetuned to new observation and action spaces. We also
perform detailed ablations of design decisions for the Octo model, from
architecture to training data, to guide future research on building generalist
robot models.
664c12dcae011af8d11e07dc
Octo Model Team
664c12dcae011af8d11e07dd
Dibya Ghosh
admin_assigned
2024-05-21T08:58:21.835Z
651b098f2930812657b90cce
/avatars/8b93a87924316866fffc6acf10a41505.svg
Dibya Ghosh
user
dibyaghosh
664c12dcae011af8d11e07de
Homer Walke
664c12dcae011af8d11e07df
Karl Pertsch
admin_assigned
2024-05-21T08:58:37.919Z
65d4c1ff29b4ac81c265e6e6
https://cdn-avatars.huggingface.co/v1/production/uploads/65d4c1ff29b4ac81c265e6e6/GXgs28okxGfpBdqhxov9-.png
Karl Pertsch
user
KarlP
664c12dcae011af8d11e07e0
Kevin Black
664c12dcae011af8d11e07e1
Oier Mees
admin_assigned
2024-05-21T08:59:25.660Z
663a7190d9dee283e3f56150
https://cdn-avatars.huggingface.co/v1/production/uploads/663a7190d9dee283e3f56150/zoYBdFIQPGWTS0R7bpg07.jpeg
Oier Mees
user
oier-mees
664c12dcae011af8d11e07e2
Sudeep Dasari
664c12dcae011af8d11e07e3
Joey Hejna
664c12dcae011af8d11e07e4
Tobias Kreiman
664c12dcae011af8d11e07e5
Charles Xu
admin_assigned
2024-05-21T09:00:26.427Z
658fdaacac02633c0d78e3e8
/avatars/c63481494f86368185a9e33587de77d3.svg
Charles Xu
user
charlesxu0124
664c12dcae011af8d11e07e6
Jianlan Luo
admin_assigned
2024-05-21T09:01:01.937Z
64f8cb8ed04a890f5380d9a4
/avatars/d6fdfdbb0c10141aa3b4c832d928121b.svg
Jianlan Luo
user
jianlanluo
664c12dcae011af8d11e07e7
You Liang Tan
admin_assigned
2024-05-21T09:01:23.423Z
635dcbee79599a4f7db46db3
https://cdn-avatars.huggingface.co/v1/production/uploads/635dcbee79599a4f7db46db3/eIi5cPQelHi0TkNNe1_XT.jpeg
You Liang Tan
user
youliangtan
664c12dcae011af8d11e07e8
Pannag Sanketi
claimed_verified
2024-08-09T07:48:53.973Z
66b56f73fc66a466e6104bfb
/avatars/74855aaf580144fafead4b93d6a24a46.svg
Pannag Sanketi
user
prsanky
664c12dcae011af8d11e07e9
Quan Vuong
admin_assigned
2024-05-21T09:01:35.171Z
63e1d6687fbb6ae4d4f4d980
/avatars/4959319e69d69e03a31ff4be326e602a.svg
Quan Vuong
user
quintonx
664c12dcae011af8d11e07ea
Ted Xiao
664c12dcae011af8d11e07eb
Dorsa Sadigh
664c12dcae011af8d11e07ec
Chelsea Finn
admin_assigned
2024-05-21T09:02:07.627Z
64ac22a9193f0a807deb673d
/avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
Chelsea Finn
user
cbfinn
664c12dcae011af8d11e07ed
Sergey Levine
Octo: An Open-Source Generalist Robot Policy
Large policies pretrained on diverse robot datasets have the potential to
transform robotic learning: instead of training new policies from scratch, such
generalist robot policies may be finetuned with only a little in-domain data,
yet generalize broadly. However, to be widely applicable across a range of
robotic learning scenarios, environments, and tasks, such policies need to
handle diverse sensors and action spaces, accommodate a variety of commonly
used robotic platforms, and finetune readily and efficiently to new domains. In
this work, we aim to lay the groundwork for developing open-source, widely
applicable, generalist policies for robotic manipulation. As a first step, we
introduce Octo, a large transformer-based policy trained on 800k trajectories
from the Open X-Embodiment dataset, the largest robot manipulation dataset to
date. It can be instructed via language commands or goal images and can be
effectively finetuned to robot setups with new sensory inputs and action spaces
within a few hours on standard consumer GPUs. In experiments across 9 robotic
platforms, we demonstrate that Octo serves as a versatile policy initialization
that can be effectively finetuned to new observation and action spaces. We also
perform detailed ablations of design decisions for the Octo model, from
architecture to training data, to guide future research on building generalist
robot models.
Octo, a large transformer-based policy trained on extensive robotic datasets, demonstrates versatility and efficient fine-tuning for diverse robotic platforms and tasks.
transformer-based policy Open X-Embodiment dataset language commands goal images fine-tuning sensory inputs action spaces ablations architecture training data generalist robot models