RACER: Rich Language-Guided Failure Recovery Policies for Imitation
  Learning
Developing robust and correctable visuomotor policies for robotic
manipulation is challenging due to the lack of self-recovery mechanisms from
failures and the limitations of simple language instructions in guiding robot
actions. To address these issues, we propose a scalable data generation
pipeline that automatically augments expert demonstrations with failure
recovery trajectories and fine-grained language annotations for training. We
then introduce Rich languAge-guided failure reCovERy (RACER), a
supervisor-actor framework, which combines failure recovery data with rich
language descriptions to enhance robot control. RACER features a
vision-language model (VLM) that acts as an online supervisor, providing
detailed language guidance for error correction and task execution, and a
language-conditioned visuomotor policy as an actor to predict the next actions.
Our experimental results show that RACER outperforms the state-of-the-art
Robotic View Transformer (RVT) on RLbench across various evaluation settings,
including standard long-horizon tasks, dynamic goal-change tasks and zero-shot
unseen tasks, achieving superior performance in both simulated and real world
environments. Videos and code are available at:
https://rich-language-failure-recovery.github.io.
66f227620fa7f61205acaa41
Yinpei Dai
claimed_verified
2024-09-24T07:52:54.065Z
648b889575d21f74acf9cbb1
/avatars/0ffee0809f9e2670821721c005d8dde4.svg
Dai
user
Yinpei
66f227620fa7f61205acaa42
Jayjun Lee
admin_assigned
2024-09-24T10:19:27.756Z
62622f39aee4bea02143fe35
/avatars/2ba33ca06a4f8aec8617c09e7f6853ce.svg
Jayjun Lee
user
newwater
66f227620fa7f61205acaa43
Nima Fazeli
admin_assigned
2024-09-24T10:19:34.083Z
66919fa1306a261f1850b27e
/avatars/34d7956986bef23d01c2bd4ebd4e9765.svg
Nima Fazeli
user
nimafazeli
66f227620fa7f61205acaa44
Joyce Chai
RACER: Rich Language-Guided Failure Recovery Policies for Imitation
  Learning
Developing robust and correctable visuomotor policies for robotic
manipulation is challenging due to the lack of self-recovery mechanisms from
failures and the limitations of simple language instructions in guiding robot
actions. To address these issues, we propose a scalable data generation
pipeline that automatically augments expert demonstrations with failure
recovery trajectories and fine-grained language annotations for training. We
then introduce Rich languAge-guided failure reCovERy (RACER), a
supervisor-actor framework, which combines failure recovery data with rich
language descriptions to enhance robot control. RACER features a
vision-language model (VLM) that acts as an online supervisor, providing
detailed language guidance for error correction and task execution, and a
language-conditioned visuomotor policy as an actor to predict the next actions.
Our experimental results show that RACER outperforms the state-of-the-art
Robotic View Transformer (RVT) on RLbench across various evaluation settings,
including standard long-horizon tasks, dynamic goal-change tasks and zero-shot
unseen tasks, achieving superior performance in both simulated and real world
environments. Videos and code are available at:
https://rich-language-failure-recovery.github.io.
supervisor-actor framework failure recovery trajectories fine-grained language annotations Rich languAge-guided failure reCovERy (RACER) vision-language model (VLM) language-guided error correction task execution language-conditioned visuomotor policy standard long-horizon tasks dynamic goal-change tasks zero-shot unseen tasks performance in simulated environments performance in real world environments Robotic View Transformer (RVT) RLbench