VoxPoser: Composable 3D Value Maps for Robotic Manipulation with
  Language Models
Large language models (LLMs) are shown to possess a wealth of actionable
knowledge that can be extracted for robot manipulation in the form of reasoning
and planning. Despite the progress, most still rely on pre-defined motion
primitives to carry out the physical interactions with the environment, which
remains a major bottleneck. In this work, we aim to synthesize robot
trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a
large variety of manipulation tasks given an open-set of instructions and an
open-set of objects. We achieve this by first observing that LLMs excel at
inferring affordances and constraints given a free-form language instruction.
More importantly, by leveraging their code-writing capabilities, they can
interact with a visual-language model (VLM) to compose 3D value maps to ground
the knowledge into the observation space of the agent. The composed value maps
are then used in a model-based planning framework to zero-shot synthesize
closed-loop robot trajectories with robustness to dynamic perturbations. We
further demonstrate how the proposed framework can benefit from online
experiences by efficiently learning a dynamics model for scenes that involve
contact-rich interactions. We present a large-scale study of the proposed
method in both simulated and real-robot environments, showcasing the ability to
perform a large variety of everyday manipulation tasks specified in free-form
natural language. Project website: https://voxposer.github.io
64af5fcd8965d8311042cc7a
Wenlong Huang
extracted_pending
2024-08-26T22:36:14.847Z
66cd0333d94b6885fd163ff7
/avatars/9117dd683fa19c1cb11f97355babcae2.svg
Wenlong Huang
user
wenlongh
64af5fcd8965d8311042cc7b
Chen Wang
64af5fcd8965d8311042cc7c
Ruohan Zhang
64af5fcd8965d8311042cc7d
Yunzhu Li
admin_assigned
2023-07-13T09:54:51.413Z
64a46562c641afb468a3dc04
/avatars/bc54e5c13f657251b05bf2725086e56d.svg
Yunzhu Li
user
yunzhuli
64af5fcd8965d8311042cc7e
Jiajun Wu
admin_assigned
2023-07-13T09:55:10.220Z
647b7b8345616c3de3627863
/avatars/8196364bffed71086ba9a0a4a62027d3.svg
Jiajun Wu
user
jiajunwu
64af5fcd8965d8311042cc7f
Li Fei-Fei
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with
  Language Models
Large language models (LLMs) are shown to possess a wealth of actionable
knowledge that can be extracted for robot manipulation in the form of reasoning
and planning. Despite the progress, most still rely on pre-defined motion
primitives to carry out the physical interactions with the environment, which
remains a major bottleneck. In this work, we aim to synthesize robot
trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a
large variety of manipulation tasks given an open-set of instructions and an
open-set of objects. We achieve this by first observing that LLMs excel at
inferring affordances and constraints given a free-form language instruction.
More importantly, by leveraging their code-writing capabilities, they can
interact with a visual-language model (VLM) to compose 3D value maps to ground
the knowledge into the observation space of the agent. The composed value maps
are then used in a model-based planning framework to zero-shot synthesize
closed-loop robot trajectories with robustness to dynamic perturbations. We
further demonstrate how the proposed framework can benefit from online
experiences by efficiently learning a dynamics model for scenes that involve
contact-rich interactions. We present a large-scale study of the proposed
method in both simulated and real-robot environments, showcasing the ability to
perform a large variety of everyday manipulation tasks specified in free-form
natural language. Project website: https://voxposer.github.io
A framework combining large language models, visual-language models, and model-based planning synthesizes robust robot trajectories for diverse manipulation tasks from natural language instructions.
large language models robot manipulation reasoning planning motion primitives 6-DoF end-effector waypoints affordances constraints code-writing capabilities visual-language model 3D value maps model-based planning closed-loop robot trajectories dynamics model contact-rich interactions large-scale study simulated environments real-robot environments