GPT-4V(ision) for Robotics: Multimodal Task Planning from Human
  Demonstration
We introduce a pipeline that enhances a general-purpose Vision Language
Model, GPT-4V(ision), by integrating observations of human actions to
facilitate robotic manipulation. This system analyzes videos of humans
performing tasks and creates executable robot programs that incorporate
affordance insights. The computation starts by analyzing the videos with GPT-4V
to convert environmental and action details into text, followed by a
GPT-4-empowered task planner. In the following analyses, vision systems
reanalyze the video with the task plan. Object names are grounded using an
open-vocabulary object detector, while focus on the hand-object relation helps
to detect the moment of grasping and releasing. This spatiotemporal grounding
allows the vision systems to further gather affordance data (e.g., grasp type,
way points, and body postures). Experiments across various scenarios
demonstrate this method's efficacy in achieving real robots' operations from
human demonstrations in a zero-shot manner. The prompts of GPT-4V/GPT-4 are
available at this project page:
https://microsoft.github.io/GPT4Vision-Robot-Manipulation-Prompts/
655c16ba71ed242b42677f8d
Naoki Wake
extracted_confirmed
2023-11-21T02:33:31.974Z
650bd16424da30e773ec74ca
/avatars/f73536f9699d3759f8df8619c3084afc.svg
Wake
user
NaokiWake
655c16ba71ed242b42677f8e
Atsushi Kanehira
655c16ba71ed242b42677f8f
Kazuhiro Sasabuchi
655c16ba71ed242b42677f90
Jun Takamatsu
655c16ba71ed242b42677f91
Katsushi Ikeuchi
GPT-4V(ision) for Robotics: Multimodal Task Planning from Human
  Demonstration
We introduce a pipeline that enhances a general-purpose Vision Language
Model, GPT-4V(ision), by integrating observations of human actions to
facilitate robotic manipulation. This system analyzes videos of humans
performing tasks and creates executable robot programs that incorporate
affordance insights. The computation starts by analyzing the videos with GPT-4V
to convert environmental and action details into text, followed by a
GPT-4-empowered task planner. In the following analyses, vision systems
reanalyze the video with the task plan. Object names are grounded using an
open-vocabulary object detector, while focus on the hand-object relation helps
to detect the moment of grasping and releasing. This spatiotemporal grounding
allows the vision systems to further gather affordance data (e.g., grasp type,
way points, and body postures). Experiments across various scenarios
demonstrate this method's efficacy in achieving real robots' operations from
human demonstrations in a zero-shot manner. The prompts of GPT-4V/GPT-4 are
available at this project page:
https://microsoft.github.io/GPT4Vision-Robot-Manipulation-Prompts/
A pipeline integrating human action observations into a general-purpose vision-language model enables robotic manipulation through task planning and spatiotemporal grounding of affordance data.
General-purpose Vision Language Model GPT-4V robotic manipulation video analysis GPT-4-empowered task planner open-vocabulary object detector hand-object relation spatiotemporal grounding affordance data zero-shot operations