PhysToolBench: Benchmarking Physical Tool Understanding for MLLMs
The ability to use, understand, and create tools is a hallmark of human
intelligence, enabling sophisticated interaction with the physical world. For
any general-purpose intelligent agent to achieve true versatility, it must also
master these fundamental skills. While modern Multimodal Large Language Models
(MLLMs) leverage their extensive common knowledge for high-level planning in
embodied AI and in downstream Vision-Language-Action (VLA) models, the extent
of their true understanding of physical tools remains unquantified. To bridge
this gap, we present PhysToolBench, the first benchmark dedicated to evaluating
the comprehension of physical tools by MLLMs. Our benchmark is structured as a
Visual Question Answering (VQA) dataset comprising over 1,000 image-text pairs.
It assesses capabilities across three distinct difficulty levels: (1) Tool
Recognition: Requiring the recognition of a tool's primary function. (2) Tool
Understanding: Testing the ability to grasp the underlying principles of a
tool's operation. (3) Tool Creation: Challenging the model to fashion a new
tool from surrounding objects when conventional options are unavailable. Our
comprehensive evaluation of 32 MLLMs-spanning proprietary, open-source,
specialized embodied, and backbones in VLAs-reveals a significant deficiency in
tool understanding. Furthermore, we provide an in-depth analysis and propose
preliminary solutions. Code and dataset are publicly available.
68ec63a6cd07fb414898c989
Zixin Zhang
68ec63a6cd07fb414898c98a
Kanghao Chen
68ec63a6cd07fb414898c98b
Xingwang Lin
68ec63a6cd07fb414898c98c
Lutao Jiang
68ec63a6cd07fb414898c98d
Xu Zheng
68ec63a6cd07fb414898c98e
Yuanhuiyi Lyu
68ec63a6cd07fb414898c98f
Litao Guo
68ec63a6cd07fb414898c990
Yinchuan Li
68ec63a6cd07fb414898c991
Ying-Cong Chen
PhysToolBench: Benchmarking Physical Tool Understanding for MLLMs
The ability to use, understand, and create tools is a hallmark of human
intelligence, enabling sophisticated interaction with the physical world. For
any general-purpose intelligent agent to achieve true versatility, it must also
master these fundamental skills. While modern Multimodal Large Language Models
(MLLMs) leverage their extensive common knowledge for high-level planning in
embodied AI and in downstream Vision-Language-Action (VLA) models, the extent
of their true understanding of physical tools remains unquantified. To bridge
this gap, we present PhysToolBench, the first benchmark dedicated to evaluating
the comprehension of physical tools by MLLMs. Our benchmark is structured as a
Visual Question Answering (VQA) dataset comprising over 1,000 image-text pairs.
It assesses capabilities across three distinct difficulty levels: (1) Tool
Recognition: Requiring the recognition of a tool's primary function. (2) Tool
Understanding: Testing the ability to grasp the underlying principles of a
tool's operation. (3) Tool Creation: Challenging the model to fashion a new
tool from surrounding objects when conventional options are unavailable. Our
comprehensive evaluation of 32 MLLMs-spanning proprietary, open-source,
specialized embodied, and backbones in VLAs-reveals a significant deficiency in
tool understanding. Furthermore, we provide an in-depth analysis and propose
preliminary solutions. Code and dataset are publicly available.
PhysToolBench evaluates MLLMs' comprehension of physical tools through a VQA dataset, revealing significant deficiencies in tool understanding.
Multimodal Large Language Models MLLMs embodied AI Vision-Language-Action VLA Visual Question Answering VQA Tool Recognition Tool Understanding Tool Creation