RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse
  Tools
Humans excel in complex long-horizon soft body manipulation tasks via
flexible tool use: bread baking requires a knife to slice the dough and a
rolling pin to flatten it. Often regarded as a hallmark of human cognition,
tool use in autonomous robots remains limited due to challenges in
understanding tool-object interactions. Here we develop an intelligent robotic
system, RoboCook, which perceives, models, and manipulates elasto-plastic
objects with various tools. RoboCook uses point cloud scene representations,
models tool-object interactions with Graph Neural Networks (GNNs), and combines
tool classification with self-supervised policy learning to devise manipulation
plans. We demonstrate that from just 20 minutes of real-world interaction data
per tool, a general-purpose robot arm can learn complex long-horizon soft
object manipulation tasks, such as making dumplings and alphabet letter
cookies. Extensive evaluations show that RoboCook substantially outperforms
state-of-the-art approaches, exhibits robustness against severe external
disturbances, and demonstrates adaptability to different materials.
649a887a685215f7248a7602
Haochen Shi
claimed_verified
2023-06-28T20:15:33.394Z
649c6f724cc1419329386141
/avatars/10efdd9a7cd4200a021bb342cd5f2481.svg
Haochen Shi
user
hshi74
649a887a685215f7248a7603
Huazhe Xu
claimed_verified
2023-06-28T20:15:26.624Z
649c130f92525b3ac9fbcffe
/avatars/b60e4d6803f7e4ef42200ddcb9e5bac1.svg
Huazhe Xu
user
Huazhe-Xu
649a887a685215f7248a7604
Samuel Clarke
649a887a685215f7248a7605
Yunzhu Li
claimed_verified
2023-07-04T19:17:32.373Z
64a46562c641afb468a3dc04
/avatars/bc54e5c13f657251b05bf2725086e56d.svg
Yunzhu Li
user
yunzhuli
649a887a685215f7248a7606
Jiajun Wu
claimed_verified
2023-06-28T20:15:29.369Z
647b7b8345616c3de3627863
/avatars/8196364bffed71086ba9a0a4a62027d3.svg
Jiajun Wu
user
jiajunwu
RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse
  Tools
Humans excel in complex long-horizon soft body manipulation tasks via
flexible tool use: bread baking requires a knife to slice the dough and a
rolling pin to flatten it. Often regarded as a hallmark of human cognition,
tool use in autonomous robots remains limited due to challenges in
understanding tool-object interactions. Here we develop an intelligent robotic
system, RoboCook, which perceives, models, and manipulates elasto-plastic
objects with various tools. RoboCook uses point cloud scene representations,
models tool-object interactions with Graph Neural Networks (GNNs), and combines
tool classification with self-supervised policy learning to devise manipulation
plans. We demonstrate that from just 20 minutes of real-world interaction data
per tool, a general-purpose robot arm can learn complex long-horizon soft
object manipulation tasks, such as making dumplings and alphabet letter
cookies. Extensive evaluations show that RoboCook substantially outperforms
state-of-the-art approaches, exhibits robustness against severe external
disturbances, and demonstrates adaptability to different materials.
point cloud scene representations Graph Neural Networks (GNNs) tool classification self-supervised policy learning elasto-plastic objects RoboCook Dumplings alphabet letter cookies