PhyX: Does Your Model Have the "Wits" for Physical Reasoning?
Existing benchmarks fail to capture a crucial aspect of intelligence:
physical reasoning, the integrated ability to combine domain knowledge,
symbolic reasoning, and understanding of real-world constraints. To address
this gap, we introduce PhyX: the first large-scale benchmark designed to assess
models capacity for physics-grounded reasoning in visual scenarios. PhyX
includes 3K meticulously curated multimodal questions spanning 6 reasoning
types across 25 sub-domains and 6 core physics domains: thermodynamics,
electromagnetism, mechanics, modern physics, optics, and wave\&acoustics. In
our comprehensive evaluation, even state-of-the-art models struggle
significantly with physical reasoning. GPT-4o, Claude3.7-Sonnet, and
GPT-o4-mini achieve only 32.5\%, 42.2\%, and 45.8\% accuracy
respectively-performance gaps exceeding 29\% compared to human experts. Our
analysis exposes critical limitations in current models: over-reliance on
memorized disciplinary knowledge, excessive dependence on mathematical
formulations, and surface-level visual pattern matching rather than genuine
physical understanding. We provide in-depth analysis through fine-grained
statistics, detailed case studies, and multiple evaluation paradigms to
thoroughly examine physical reasoning capabilities. To ensure reproducibility,
we implement a compatible evaluation protocol based on widely-used toolkits
such as VLMEvalKit, enabling one-click evaluation.
6830404effb59afb6569273a
Hui Shen
claimed_verified
2025-05-27T07:55:15.229Z
640feb91b0ee289c8583dd59
https://cdn-avatars.huggingface.co/v1/production/uploads/640feb91b0ee289c8583dd59/CA4UZj2mEsJO2UR0NnN5H.jpeg
Hui Shen
user
Cloudriver
6830404effb59afb6569273b
Taiqiang Wu
claimed_verified
2025-05-26T08:14:30.851Z
6621cea88850e38ffbb1854f
https://cdn-avatars.huggingface.co/v1/production/uploads/6621cea88850e38ffbb1854f/LeytEEjSwnnqB-zFN1Tgt.jpeg
Taki WU
user
taki555
6830404effb59afb6569273c
Qi Han
claimed_verified
2025-05-27T07:55:12.818Z
6766d2764d57021c96679b7f
/avatars/0810011e1e0636cb80eb7c092ea13b3f.svg
han q
user
nrzyh
6830404effb59afb6569273d
Yunta Hsieh
6830404effb59afb6569273e
Jizhou Wang
claimed_verified
2025-05-26T08:14:33.171Z
67fe265f9698ae4f5f4db718
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/0PLJEbUyJbM9BacFxcScP.png
Jizhou Wang
user
John-ai-bee
6830404effb59afb6569273f
Yuyue Zhang
6830404effb59afb65692740
Yuxin Cheng
6830404effb59afb65692741
Zijian Hao
6830404effb59afb65692742
Yuansheng Ni
claimed_verified
2025-05-28T09:01:10.443Z
64de37ee5e192985054be575
https://cdn-avatars.huggingface.co/v1/production/uploads/64de37ee5e192985054be575/fVV7JQMtp_J3uFqszJJHH.jpeg
Yuansheng Ni
user
yuanshengni
6830404effb59afb65692743
Xin Wang
6830404effb59afb65692744
Zhongwei Wan
6830404effb59afb65692745
Kai Zhang
6830404effb59afb65692746
Wendong Xu
6830404effb59afb65692747
Jing Xiong
6830404effb59afb65692748
Ping Luo
6830404effb59afb65692749
Wenhu Chen
6830404effb59afb6569274a
Chaofan Tao
6830404effb59afb6569274b
Zhuoqing Mao
6830404effb59afb6569274c
Ngai Wong
PhyX: Does Your Model Have the "Wits" for Physical Reasoning?
Existing benchmarks fail to capture a crucial aspect of intelligence:
physical reasoning, the integrated ability to combine domain knowledge,
symbolic reasoning, and understanding of real-world constraints. To address
this gap, we introduce PhyX: the first large-scale benchmark designed to assess
models capacity for physics-grounded reasoning in visual scenarios. PhyX
includes 3K meticulously curated multimodal questions spanning 6 reasoning
types across 25 sub-domains and 6 core physics domains: thermodynamics,
electromagnetism, mechanics, modern physics, optics, and wave\&acoustics. In
our comprehensive evaluation, even state-of-the-art models struggle
significantly with physical reasoning. GPT-4o, Claude3.7-Sonnet, and
GPT-o4-mini achieve only 32.5\%, 42.2\%, and 45.8\% accuracy
respectively-performance gaps exceeding 29\% compared to human experts. Our
analysis exposes critical limitations in current models: over-reliance on
memorized disciplinary knowledge, excessive dependence on mathematical
formulations, and surface-level visual pattern matching rather than genuine
physical understanding. We provide in-depth analysis through fine-grained
statistics, detailed case studies, and multiple evaluation paradigms to
thoroughly examine physical reasoning capabilities. To ensure reproducibility,
we implement a compatible evaluation protocol based on widely-used toolkits
such as VLMEvalKit, enabling one-click evaluation.
A new benchmark, PhyX, evaluates models' physics-grounded reasoning in visual scenarios, revealing significant limitations in current models' physical understanding compared to human experts.
multimodal questions reasoning types sub-domains core physics domains thermodynamics electromagnetism mechanics modern physics optics wave\&acoustics fine-grained statistics case studies evaluation paradigms VLMEvalKit