KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for
  Open-Vocabulary Robotic Manipulation
With the rapid advancement of large language models (LLMs) and
vision-language models (VLMs), significant progress has been made in developing
open-vocabulary robotic manipulation systems. However, many existing approaches
overlook the importance of object dynamics, limiting their applicability to
more complex, dynamic tasks. In this work, we introduce KUDA, an
open-vocabulary manipulation system that integrates dynamics learning and
visual prompting through keypoints, leveraging both VLMs and learning-based
neural dynamics models. Our key insight is that a keypoint-based target
specification is simultaneously interpretable by VLMs and can be efficiently
translated into cost functions for model-based planning. Given language
instructions and visual observations, KUDA first assigns keypoints to the RGB
image and queries the VLM to generate target specifications. These abstract
keypoint-based representations are then converted into cost functions, which
are optimized using a learned dynamics model to produce robotic trajectories.
We evaluate KUDA on a range of manipulation tasks, including free-form language
instructions across diverse object categories, multi-object interactions, and
deformable or granular objects, demonstrating the effectiveness of our
framework. The project page is available at http://kuda-dynamics.github.io.
67da141d6b2857e3ec1412a7
Zixian Liu
67da141d6b2857e3ec1412a8
Mingtong Zhang
admin_assigned
2025-03-19T13:46:28.495Z
671c6a3e255aa50ebb504fc5
/avatars/b549bb19990ae21690472799817f951e.svg
Mingtong Zhang
user
Mingtongz
67da141d6b2857e3ec1412a9
Yunzhu Li
admin_assigned
2025-03-19T13:46:19.486Z
64a46562c641afb468a3dc04
/avatars/bc54e5c13f657251b05bf2725086e56d.svg
Yunzhu Li
user
yunzhuli
KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for
  Open-Vocabulary Robotic Manipulation
With the rapid advancement of large language models (LLMs) and
vision-language models (VLMs), significant progress has been made in developing
open-vocabulary robotic manipulation systems. However, many existing approaches
overlook the importance of object dynamics, limiting their applicability to
more complex, dynamic tasks. In this work, we introduce KUDA, an
open-vocabulary manipulation system that integrates dynamics learning and
visual prompting through keypoints, leveraging both VLMs and learning-based
neural dynamics models. Our key insight is that a keypoint-based target
specification is simultaneously interpretable by VLMs and can be efficiently
translated into cost functions for model-based planning. Given language
instructions and visual observations, KUDA first assigns keypoints to the RGB
image and queries the VLM to generate target specifications. These abstract
keypoint-based representations are then converted into cost functions, which
are optimized using a learned dynamics model to produce robotic trajectories.
We evaluate KUDA on a range of manipulation tasks, including free-form language
instructions across diverse object categories, multi-object interactions, and
deformable or granular objects, demonstrating the effectiveness of our
framework. The project page is available at http://kuda-dynamics.github.io.
LLMs VLMs open-vocabulary robotic manipulation systems object dynamics KUDA dynamics learning visual prompting keypoints learning-based neural dynamics models keypoint-based target specification cost functions model-based planning robotic trajectories free-form language instructions multi-object interactions deformable objects granular objects