Physically Grounded Vision-Language Models for Robotic Manipulation
Recent advances in vision-language models (VLMs) have led to improved
performance on tasks such as visual question answering and image captioning.
Consequently, these models are now well-positioned to reason about the physical
world, particularly within domains such as robotic manipulation. However,
current VLMs are limited in their understanding of the physical concepts (e.g.,
material, fragility) of common objects, which restricts their usefulness for
robotic manipulation tasks that involve interaction and physical reasoning
about such objects. To address this limitation, we propose PhysObjects, an
object-centric dataset of 36.9K crowd-sourced and 417K automated physical
concept annotations of common household objects. We demonstrate that
fine-tuning a VLM on PhysObjects improves its understanding of physical object
concepts, by capturing human priors of these concepts from visual appearance.
We incorporate this physically-grounded VLM in an interactive framework with a
large language model-based robotic planner, and show improved planning
performance on tasks that require reasoning about physical object concepts,
compared to baselines that do not leverage physically-grounded VLMs. We
additionally illustrate the benefits of our physically-grounded VLM on a real
robot, where it improves task success rates. We release our dataset and provide
further details and visualizations of our results at
https://iliad.stanford.edu/pg-vlm/.
64f92c90fe57e7455ac3ed2d
Jensen Gao
admin_assigned
2023-09-07T13:30:30.738Z
64adc155cf90fe2755648257
/avatars/a449c4f5bc051b1e4bd0441667f4c2f8.svg
Jensen Gao
user
jenseng
64f92c90fe57e7455ac3ed2e
Bidipta Sarkar
admin_assigned
2023-09-07T13:30:37.971Z
63abbf74ad514ca8d14a0548
/avatars/b1357b73b8f9a8ff9908710ad64154ef.svg
Bidipta Sarkar
user
bidiptas
64f92c90fe57e7455ac3ed2f
Fei Xia
claimed_verified
2024-01-23T08:10:34.384Z
62cfa9ee9b96f22525c36d47
/avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
Fei Xia
user
fxia22
64f92c90fe57e7455ac3ed30
Ted Xiao
extracted_pending
2025-03-06T07:26:16.159Z
64f1835d8eaa168d21f7bac4
/avatars/59754ba46756ecdab8d7dff740af90a6.svg
Ted Xiao
user
xiaoted
64f92c90fe57e7455ac3ed31
Jiajun Wu
admin_assigned
2023-09-07T13:32:03.787Z
647b7b8345616c3de3627863
/avatars/8196364bffed71086ba9a0a4a62027d3.svg
Jiajun Wu
user
jiajunwu
64f92c90fe57e7455ac3ed32
Brian Ichter
admin_assigned
2023-09-07T13:32:11.167Z
633a26ab474cfeb1a864dc56
/avatars/cd40accd894fc78810fc0d5108f413e9.svg
Brian I
user
brianichter
64f92c90fe57e7455ac3ed33
Anirudha Majumdar
64f92c90fe57e7455ac3ed34
Dorsa Sadigh
Physically Grounded Vision-Language Models for Robotic Manipulation
Recent advances in vision-language models (VLMs) have led to improved
performance on tasks such as visual question answering and image captioning.
Consequently, these models are now well-positioned to reason about the physical
world, particularly within domains such as robotic manipulation. However,
current VLMs are limited in their understanding of the physical concepts (e.g.,
material, fragility) of common objects, which restricts their usefulness for
robotic manipulation tasks that involve interaction and physical reasoning
about such objects. To address this limitation, we propose PhysObjects, an
object-centric dataset of 36.9K crowd-sourced and 417K automated physical
concept annotations of common household objects. We demonstrate that
fine-tuning a VLM on PhysObjects improves its understanding of physical object
concepts, by capturing human priors of these concepts from visual appearance.
We incorporate this physically-grounded VLM in an interactive framework with a
large language model-based robotic planner, and show improved planning
performance on tasks that require reasoning about physical object concepts,
compared to baselines that do not leverage physically-grounded VLMs. We
additionally illustrate the benefits of our physically-grounded VLM on a real
robot, where it improves task success rates. We release our dataset and provide
further details and visualizations of our results at
https://iliad.stanford.edu/pg-vlm/.
A physically-grounded vision-language model (VLM) fine-tuned on a dataset of object physical properties improves robotic manipulation and planning performance.
vision-language models VLMs visual question answering image captioning robotic manipulation physical concepts object-centric dataset fine-tuning human priors large language model-based robotic planner task success rates