Goal Representations for Instruction Following: A Semi-Supervised
  Language Interface to Control
Our goal is for robots to follow natural language instructions like "put the
towel next to the microwave." But getting large amounts of labeled data, i.e.
data that contains demonstrations of tasks labeled with the language
instruction, is prohibitive. In contrast, obtaining policies that respond to
image goals is much easier, because any autonomous trial or demonstration can
be labeled in hindsight with its final state as the goal. In this work, we
contribute a method that taps into joint image- and goal- conditioned policies
with language using only a small amount of language data. Prior work has made
progress on this using vision-language models or by jointly training
language-goal-conditioned policies, but so far neither method has scaled
effectively to real-world robot tasks without significant human annotation. Our
method achieves robust performance in the real world by learning an embedding
from the labeled data that aligns language not to the goal image, but rather to
the desired change between the start and goal images that the instruction
corresponds to. We then train a policy on this embedding: the policy benefits
from all the unlabeled data, but the aligned embedding provides an interface
for language to steer the policy. We show instruction following across a
variety of manipulation tasks in different scenes, with generalization to
language instructions outside of the labeled data. Videos and code for our
approach can be found on our website: http://tiny.cc/grif .
64a37def6badc2ebd1dde7be
Vivek Myers
64a37def6badc2ebd1dde7bf
Andre He
64a37def6badc2ebd1dde7c0
Kuan Fang
64a37def6badc2ebd1dde7c1
Homer Walke
64a37def6badc2ebd1dde7c2
Philippe Hansen-Estruch
claimed_verified
2024-12-17T08:05:49.347Z
644eb923a8f384abd038a4f7
/avatars/3d1a29c6149a6ffc482842f1438ba55a.svg
Philippe I Hansen-Estruch
user
philippehansen
64a37def6badc2ebd1dde7c3
Ching-An Cheng
64a37def6badc2ebd1dde7c4
Mihai Jalobeanu
64a37def6badc2ebd1dde7c5
Andrey Kolobov
claimed_verified
2023-07-04T19:17:12.661Z
6350478dc5acdbefb8ecf722
/avatars/75f3cdc38f6e67ae0435c30aff50b089.svg
Andrey Kolobov
user
akolobov
64a37def6badc2ebd1dde7c6
Anca Dragan
64a37def6badc2ebd1dde7c7
Sergey Levine
Goal Representations for Instruction Following: A Semi-Supervised
  Language Interface to Control
Our goal is for robots to follow natural language instructions like "put the
towel next to the microwave." But getting large amounts of labeled data, i.e.
data that contains demonstrations of tasks labeled with the language
instruction, is prohibitive. In contrast, obtaining policies that respond to
image goals is much easier, because any autonomous trial or demonstration can
be labeled in hindsight with its final state as the goal. In this work, we
contribute a method that taps into joint image- and goal- conditioned policies
with language using only a small amount of language data. Prior work has made
progress on this using vision-language models or by jointly training
language-goal-conditioned policies, but so far neither method has scaled
effectively to real-world robot tasks without significant human annotation. Our
method achieves robust performance in the real world by learning an embedding
from the labeled data that aligns language not to the goal image, but rather to
the desired change between the start and goal images that the instruction
corresponds to. We then train a policy on this embedding: the policy benefits
from all the unlabeled data, but the aligned embedding provides an interface
for language to steer the policy. We show instruction following across a
variety of manipulation tasks in different scenes, with generalization to
language instructions outside of the labeled data. Videos and code for our
approach can be found on our website: http://tiny.cc/grif .
A method is introduced to enable robots to follow natural language instructions with minimal labeled data by aligning language to the change between images rather than the goal image, enhancing policy training and generalization.
vision-language models language-goal-conditioned policies embedding instruction following manipulation tasks