In-Context Imitation Learning via Next-Token Prediction
We explore how to enhance next-token prediction models to perform in-context
imitation learning on a real robot, where the robot executes new tasks by
interpreting contextual information provided during the input phase, without
updating its underlying policy parameters. We propose In-Context Robot
Transformer (ICRT), a causal transformer that performs autoregressive
prediction on sensorimotor trajectories without relying on any linguistic data
or reward function. This formulation enables flexible and training-free
execution of new tasks at test time, achieved by prompting the model with
sensorimotor trajectories of the new task composing of image observations,
actions and states tuples, collected through human teleoperation. Experiments
with a Franka Emika robot demonstrate that the ICRT can adapt to new tasks
specified by prompts, even in environment configurations that differ from both
the prompt and the training data. In a multitask environment setup, ICRT
significantly outperforms current state-of-the-art next-token prediction models
in robotics on generalizing to unseen tasks. Code, checkpoints and data are
available on https://icrt.dev/
66cfe2810c8862832e552b36
Letian Fu
admin_assigned
2024-08-29T07:53:10.487Z
64908d6b0c18343a09361b93
/avatars/2fa44344fe760f68f19de775b76f45b8.svg
Max (Letian) Fu
user
mlfu7
66cfe2810c8862832e552b37
Huang Huang
66cfe2810c8862832e552b38
Gaurav Datta
admin_assigned
2024-08-29T07:54:05.965Z
65caf5f998a46a5626426bd1
/avatars/636301ed1f3577b85787959651b93279.svg
Gaurav Datta
user
gdatta
66cfe2810c8862832e552b39
Lawrence Yunliang Chen
admin_assigned
2024-08-29T07:54:46.288Z
6683696648edefb453aa39f1
/avatars/b056d8499420966049b40bb6a03a8782.svg
Lawrence Chen
user
yunliangchen
66cfe2810c8862832e552b3a
William Chung-Ho Panitch
66cfe2810c8862832e552b3b
Fangchen Liu
admin_assigned
2024-08-29T07:55:11.431Z
63609ef63605bd411c1c7509
/avatars/3fadc3c4117cdf17d25f3efc545388cd.svg
Fangchen Liu
user
fangchenliu
66cfe2810c8862832e552b3c
Hui Li
66cfe2810c8862832e552b3d
Ken Goldberg
In-Context Imitation Learning via Next-Token Prediction
We explore how to enhance next-token prediction models to perform in-context
imitation learning on a real robot, where the robot executes new tasks by
interpreting contextual information provided during the input phase, without
updating its underlying policy parameters. We propose In-Context Robot
Transformer (ICRT), a causal transformer that performs autoregressive
prediction on sensorimotor trajectories without relying on any linguistic data
or reward function. This formulation enables flexible and training-free
execution of new tasks at test time, achieved by prompting the model with
sensorimotor trajectories of the new task composing of image observations,
actions and states tuples, collected through human teleoperation. Experiments
with a Franka Emika robot demonstrate that the ICRT can adapt to new tasks
specified by prompts, even in environment configurations that differ from both
the prompt and the training data. In a multitask environment setup, ICRT
significantly outperforms current state-of-the-art next-token prediction models
in robotics on generalizing to unseen tasks. Code, checkpoints and data are
available on https://icrt.dev/
In-Context Robot Transformer (ICRT) enables flexible and training-free execution of new tasks by interpreting sensorimotor trajectories during input phase without updating policy parameters.
next-token prediction models in-context imitation learning causal transformer autoregressive prediction sensorimotor trajectories prompt-based human teleoperation multitask environment unseen tasks