OK-Robot: What Really Matters in Integrating Open-Knowledge Models for
  Robotics
Remarkable progress has been made in recent years in the fields of vision,
language, and robotics. We now have vision models capable of recognizing
objects based on language queries, navigation systems that can effectively
control mobile systems, and grasping models that can handle a wide range of
objects. Despite these advancements, general-purpose applications of robotics
still lag behind, even though they rely on these fundamental capabilities of
recognition, navigation, and grasping. In this paper, we adopt a systems-first
approach to develop a new Open Knowledge-based robotics framework called
OK-Robot. By combining Vision-Language Models (VLMs) for object detection,
navigation primitives for movement, and grasping primitives for object
manipulation, OK-Robot offers a integrated solution for pick-and-drop
operations without requiring any training. To evaluate its performance, we run
OK-Robot in 10 real-world home environments. The results demonstrate that
OK-Robot achieves a 58.5% success rate in open-ended pick-and-drop tasks,
representing a new state-of-the-art in Open Vocabulary Mobile Manipulation
(OVMM) with nearly 1.8x the performance of prior work. On cleaner, uncluttered
environments, OK-Robot's performance increases to 82%. However, the most
important insight gained from OK-Robot is the critical role of nuanced details
when combining Open Knowledge systems like VLMs with robotic modules. Videos of
our experiments are available on our website: https://ok-robot.github.io
65af3fa852e1b2aae41667a7
Peiqi Liu
admin_assigned
2024-01-23T10:53:51.316Z
631f1793c1a8269da393eb15
/avatars/3cb07f9ef3f7f1298da7a880661b6689.svg
Peiqi Liu
user
LiuPeiqi
65af3fa852e1b2aae41667a8
Yaswanth Orru
admin_assigned
2024-01-23T10:53:58.227Z
655c6db7419819fec2cf85b2
/avatars/7de3a409d15fd60b44032bceadf2c74a.svg
Yaswanth Orru
user
yaswantho
65af3fa852e1b2aae41667a9
Chris Paxton
admin_assigned
2024-01-23T10:54:04.886Z
62b13b28b7053ab38d2d5f96
https://cdn-avatars.huggingface.co/v1/production/uploads/62b13b28b7053ab38d2d5f96/V6780Omima7sNW5E4TEeW.png
Chris Paxton
user
cpaxton
65af3fa852e1b2aae41667aa
Nur Muhammad Mahi Shafiullah
claimed_verified
2024-01-23T08:08:54.262Z
630e567f8df86f1e5bf0d837
https://cdn-avatars.huggingface.co/v1/production/uploads/630e567f8df86f1e5bf0d837/Mje-2ahgmMET2jzqLafhy.jpeg
Mahi Shafiullah
user
notmahi
65af3fa852e1b2aae41667ab
Lerrel Pinto
admin_assigned
2024-01-23T10:54:11.294Z
6388cdb6685efaedd15dc0f3
/avatars/b2b74454bf86bf61fd90ba088644f547.svg
Lerrel Pinto
user
lerrel
OK-Robot: What Really Matters in Integrating Open-Knowledge Models for
  Robotics
Remarkable progress has been made in recent years in the fields of vision,
language, and robotics. We now have vision models capable of recognizing
objects based on language queries, navigation systems that can effectively
control mobile systems, and grasping models that can handle a wide range of
objects. Despite these advancements, general-purpose applications of robotics
still lag behind, even though they rely on these fundamental capabilities of
recognition, navigation, and grasping. In this paper, we adopt a systems-first
approach to develop a new Open Knowledge-based robotics framework called
OK-Robot. By combining Vision-Language Models (VLMs) for object detection,
navigation primitives for movement, and grasping primitives for object
manipulation, OK-Robot offers a integrated solution for pick-and-drop
operations without requiring any training. To evaluate its performance, we run
OK-Robot in 10 real-world home environments. The results demonstrate that
OK-Robot achieves a 58.5% success rate in open-ended pick-and-drop tasks,
representing a new state-of-the-art in Open Vocabulary Mobile Manipulation
(OVMM) with nearly 1.8x the performance of prior work. On cleaner, uncluttered
environments, OK-Robot's performance increases to 82%. However, the most
important insight gained from OK-Robot is the critical role of nuanced details
when combining Open Knowledge systems like VLMs with robotic modules. Videos of
our experiments are available on our website: https://ok-robot.github.io
OK-Robot, a new framework combining Vision-Language Models, navigation primitives, and grasping primitives, achieves high success rates in open-ended pick-and-drop tasks across various home environments.
Vision-Language Models object detection navigation primitives grasping primitives Open Vocabulary Mobile Manipulation open-ended pick-and-drop tasks