Robot Learning with Sensorimotor Pre-training
We present a self-supervised sensorimotor pre-training approach for robotics.
Our model, called RPT, is a Transformer that operates on sequences of
sensorimotor tokens. Given a sequence of camera images, proprioceptive robot
states, and past actions, we encode the interleaved sequence into tokens, mask
out a random subset, and train a model to predict the masked-out content. We
hypothesize that if the robot can predict the missing content it has acquired a
good model of the physical world that can enable it to act. RPT is designed to
operate on latent visual representations which makes prediction tractable,
enables scaling to 10x larger models, and 10 Hz inference on a real robot. To
evaluate our approach, we collect a dataset of 20,000 real-world trajectories
over 9 months using a combination of motion planning and model-based grasping
algorithms. We find that pre-training on this data consistently outperforms
training from scratch, leads to 2x improvements in the block stacking task, and
has favorable scaling properties.
648fd8841ba574b69dfa0e10
Ilija Radosavovic
648fd8841ba574b69dfa0e11
Baifeng Shi
claimed_verified
2023-06-19T11:44:38.643Z
649004218f7cbbc94c782db6
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AdgLVfAIpWlug4jXTaEK-.jpeg
Baifeng Shi
user
bfshi
648fd8841ba574b69dfa0e12
Letian Fu
claimed_verified
2023-06-19T19:44:31.589Z
64908d6b0c18343a09361b93
/avatars/2fa44344fe760f68f19de775b76f45b8.svg
Max (Letian) Fu
user
mlfu7
648fd8841ba574b69dfa0e13
Ken Goldberg
648fd8841ba574b69dfa0e14
Trevor Darrell
648fd8841ba574b69dfa0e15
Jitendra Malik
Robot Learning with Sensorimotor Pre-training
We present a self-supervised sensorimotor pre-training approach for robotics.
Our model, called RPT, is a Transformer that operates on sequences of
sensorimotor tokens. Given a sequence of camera images, proprioceptive robot
states, and past actions, we encode the interleaved sequence into tokens, mask
out a random subset, and train a model to predict the masked-out content. We
hypothesize that if the robot can predict the missing content it has acquired a
good model of the physical world that can enable it to act. RPT is designed to
operate on latent visual representations which makes prediction tractable,
enables scaling to 10x larger models, and 10 Hz inference on a real robot. To
evaluate our approach, we collect a dataset of 20,000 real-world trajectories
over 9 months using a combination of motion planning and model-based grasping
algorithms. We find that pre-training on this data consistently outperforms
training from scratch, leads to 2x improvements in the block stacking task, and
has favorable scaling properties.
A self-supervised sensorimotor pre-training method using a Transformer model on sequences of sensorimotor tokens improves robotic performance in tasks like block stacking.
Transformer sensorimotor tokens camera images proprioceptive states past actions latent visual representations