KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for
  Open-Vocabulary Robotic Manipulation
With the rapid advancement of large language models (LLMs) and
vision-language models (VLMs), significant progress has been made in developing
open-vocabulary robotic manipulation systems. However, many existing approaches
overlook the importance of object dynamics, limiting their applicability to
more complex, dynamic tasks. In this work, we introduce KUDA, an
open-vocabulary manipulation system that integrates dynamics learning and
visual prompting through keypoints, leveraging both VLMs and learning-based
neural dynamics models. Our key insight is that a keypoint-based target
specification is simultaneously interpretable by VLMs and can be efficiently
translated into cost functions for model-based planning. Given language
instructions and visual observations, KUDA first assigns keypoints to the RGB
image and queries the VLM to generate target specifications. These abstract
keypoint-based representations are then converted into cost functions, which
are optimized using a learned dynamics model to produce robotic trajectories.
We evaluate KUDA on a range of manipulation tasks, including free-form language
instructions across diverse object categories, multi-object interactions, and
deformable or granular objects, demonstrating the effectiveness of our
framework. The project page is available at http://kuda-dynamics.github.io.
67da141d6b2857e3ec1412a7
Zixian Liu
67da141d6b2857e3ec1412a8
Mingtong Zhang
admin_assigned
2025-03-19T13:46:28.495Z
671c6a3e255aa50ebb504fc5
/avatars/b549bb19990ae21690472799817f951e.svg
Mingtong Zhang
user
Mingtongz
67da141d6b2857e3ec1412a9
Yunzhu Li
admin_assigned
2025-03-19T13:46:19.486Z
64a46562c641afb468a3dc04
/avatars/bc54e5c13f657251b05bf2725086e56d.svg
Yunzhu Li
user
yunzhuli
KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for
  Open-Vocabulary Robotic Manipulation
With the rapid advancement of large language models (LLMs) and
vision-language models (VLMs), significant progress has been made in developing
open-vocabulary robotic manipulation systems. However, many existing approaches
overlook the importance of object dynamics, limiting their applicability to
more complex, dynamic tasks. In this work, we introduce KUDA, an
open-vocabulary manipulation system that integrates dynamics learning and
visual prompting through keypoints, leveraging both VLMs and learning-based
neural dynamics models. Our key insight is that a keypoint-based target
specification is simultaneously interpretable by VLMs and can be efficiently
translated into cost functions for model-based planning. Given language
instructions and visual observations, KUDA first assigns keypoints to the RGB
image and queries the VLM to generate target specifications. These abstract
keypoint-based representations are then converted into cost functions, which
are optimized using a learned dynamics model to produce robotic trajectories.
We evaluate KUDA on a range of manipulation tasks, including free-form language
instructions across diverse object categories, multi-object interactions, and
deformable or granular objects, demonstrating the effectiveness of our
framework. The project page is available at http://kuda-dynamics.github.io.
KUDA integrates dynamics learning and visual prompting through keypoints to enhance robotic manipulation systems, enabling interpretation of language instructions and planning of robotic trajectories with cost functions derived from VLMs and neural dynamics models.
open-vocabulary robotic manipulation systems large language models vision-language models dynamics learning visual prompting keypoints target specifications model-based planning cost functions learned dynamics model robotic trajectories free-form language instructions multi-object interactions deformable objects granular objects