PhysBench: Benchmarking and Enhancing Vision-Language Models for
  Physical World Understanding
Understanding the physical world is a fundamental challenge in embodied AI,
critical for enabling agents to perform complex tasks and operate safely in
real-world environments. While Vision-Language Models (VLMs) have shown great
promise in reasoning and task planning for embodied agents, their ability to
comprehend physical phenomena remains extremely limited. To close this gap, we
introduce PhysBench, a comprehensive benchmark designed to evaluate VLMs'
physical world understanding capability across a diverse set of tasks.
PhysBench contains 10,002 entries of interleaved video-image-text data,
categorized into four major domains: physical object properties, physical
object relationships, physical scene understanding, and physics-based dynamics,
further divided into 19 subclasses and 8 distinct capability dimensions. Our
extensive experiments, conducted on 75 representative VLMs, reveal that while
these models excel in common-sense reasoning, they struggle with understanding
the physical world -- likely due to the absence of physical knowledge in their
training data and the lack of embedded physical priors. To tackle the
shortfall, we introduce PhysAgent, a novel framework that combines the
generalization strengths of VLMs with the specialized expertise of vision
models, significantly enhancing VLMs' physical understanding across a variety
of tasks, including an 18.4\% improvement on GPT-4o. Furthermore, our results
demonstrate that enhancing VLMs' physical world understanding capabilities can
help embodied agents such as MOKA. We believe that PhysBench and PhysAgent
offer valuable insights and contribute to bridging the gap between VLMs and
physical world understanding.
679c4f344061a1ab60ebe6fa
Wei Chow
claimed_verified
2025-01-31T08:35:49.674Z
644b71ddb2e7823a76abcf91
https://cdn-avatars.huggingface.co/v1/production/uploads/644b71ddb2e7823a76abcf91/JPF7Eqeq2jx8i79nQ962K.jpeg
zhou wei
user
WeiChow
679c4f344061a1ab60ebe6fb
Jiageng Mao
679c4f344061a1ab60ebe6fc
Boyi Li
admin_assigned
2025-01-31T08:46:09.305Z
620dd3888528f797e88cb9b5
/avatars/af04728788d78fe7d6375e19e32a535e.svg
Boyi Li
user
Boyiliee
679c4f344061a1ab60ebe6fd
Daniel Seita
679c4f344061a1ab60ebe6fe
Vitor Guizilini
679c4f344061a1ab60ebe6ff
Yue Wang
PhysBench: Benchmarking and Enhancing Vision-Language Models for
  Physical World Understanding
Understanding the physical world is a fundamental challenge in embodied AI,
critical for enabling agents to perform complex tasks and operate safely in
real-world environments. While Vision-Language Models (VLMs) have shown great
promise in reasoning and task planning for embodied agents, their ability to
comprehend physical phenomena remains extremely limited. To close this gap, we
introduce PhysBench, a comprehensive benchmark designed to evaluate VLMs'
physical world understanding capability across a diverse set of tasks.
PhysBench contains 10,002 entries of interleaved video-image-text data,
categorized into four major domains: physical object properties, physical
object relationships, physical scene understanding, and physics-based dynamics,
further divided into 19 subclasses and 8 distinct capability dimensions. Our
extensive experiments, conducted on 75 representative VLMs, reveal that while
these models excel in common-sense reasoning, they struggle with understanding
the physical world -- likely due to the absence of physical knowledge in their
training data and the lack of embedded physical priors. To tackle the
shortfall, we introduce PhysAgent, a novel framework that combines the
generalization strengths of VLMs with the specialized expertise of vision
models, significantly enhancing VLMs' physical understanding across a variety
of tasks, including an 18.4\% improvement on GPT-4o. Furthermore, our results
demonstrate that enhancing VLMs' physical world understanding capabilities can
help embodied agents such as MOKA. We believe that PhysBench and PhysAgent
offer valuable insights and contribute to bridging the gap between VLMs and
physical world understanding.
A benchmark and framework, PhysBench and PhysAgent, enhance Vision-Language Models' understanding of physical phenomena in real-world environments.
Vision-Language Models VLMs PhysBench PhysAgent physical object properties physical object relationships physical scene understanding physics-based dynamics vision models embodied agents MOKA