Learning to Grasp Anything by Playing with Random Toys
Robotic manipulation policies often struggle to generalize to novel objects,
limiting their real-world utility. In contrast, cognitive science suggests that
children develop generalizable dexterous manipulation skills by mastering a
small set of simple toys and then applying that knowledge to more complex
items. Inspired by this, we study if similar generalization capabilities can
also be achieved by robots. Our results indicate robots can learn generalizable
grasping using randomly assembled objects that are composed from just four
shape primitives: spheres, cuboids, cylinders, and rings. We show that training
on these "toys" enables robust generalization to real-world objects, yielding
strong zero-shot performance. Crucially, we find the key to this generalization
is an object-centric visual representation induced by our proposed detection
pooling mechanism. Evaluated in both simulation and on physical robots, our
model achieves a 67% real-world grasping success rate on the YCB dataset,
outperforming state-of-the-art approaches that rely on substantially more
in-domain data. We further study how zero-shot generalization performance
scales by varying the number and diversity of training toys and the
demonstrations per toy. We believe this work offers a promising path to
scalable and generalizable learning in robotic manipulation. Demonstration
videos, code, checkpoints and our dataset are available on our project page:
https://lego-grasp.github.io/ .
68f127eadf97aa3a74268631
Dantong Niu
68f127eadf97aa3a74268632
Yuvan Sharma
68f127eadf97aa3a74268633
Baifeng Shi
68f127eadf97aa3a74268634
Rachel Ding
68f127eadf97aa3a74268635
Matteo Gioia
68f127eadf97aa3a74268636
Haoru Xue
68f127eadf97aa3a74268637
Henry Tsai
68f127eadf97aa3a74268638
Konstantinos Kallidromitis
68f127eadf97aa3a74268639
Anirudh Pai
68f127eadf97aa3a7426863a
Shankar Shastry
68f127eadf97aa3a7426863b
Trevor Darrell
68f127eadf97aa3a7426863c
Jitendra Malik
68f127eadf97aa3a7426863d
Roei Herzig
Learning to Grasp Anything by Playing with Random Toys
Robotic manipulation policies often struggle to generalize to novel objects,
limiting their real-world utility. In contrast, cognitive science suggests that
children develop generalizable dexterous manipulation skills by mastering a
small set of simple toys and then applying that knowledge to more complex
items. Inspired by this, we study if similar generalization capabilities can
also be achieved by robots. Our results indicate robots can learn generalizable
grasping using randomly assembled objects that are composed from just four
shape primitives: spheres, cuboids, cylinders, and rings. We show that training
on these "toys" enables robust generalization to real-world objects, yielding
strong zero-shot performance. Crucially, we find the key to this generalization
is an object-centric visual representation induced by our proposed detection
pooling mechanism. Evaluated in both simulation and on physical robots, our
model achieves a 67% real-world grasping success rate on the YCB dataset,
outperforming state-of-the-art approaches that rely on substantially more
in-domain data. We further study how zero-shot generalization performance
scales by varying the number and diversity of training toys and the
demonstrations per toy. We believe this work offers a promising path to
scalable and generalizable learning in robotic manipulation. Demonstration
videos, code, checkpoints and our dataset are available on our project page:
https://lego-grasp.github.io/ .
Robots can achieve generalizable grasping skills by learning from a small set of simple objects, using an object-centric visual representation, which outperforms state-of-the-art methods with less data.
object-centric visual representation detection pooling mechanism zero-shot performance YCB dataset robotic manipulation generalizable grasping