SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics
  Reasoning
We present SeePhys, a large-scale multimodal benchmark for LLM reasoning
grounded in physics questions ranging from middle school to PhD qualifying
exams. The benchmark covers 7 fundamental domains spanning the physics
discipline, incorporating 21 categories of highly heterogeneous diagrams. In
contrast to prior works where visual elements mainly serve auxiliary purposes,
our benchmark features a substantial proportion of vision-essential problems
(75\%) that mandate visual information extraction for correct solutions.
Through extensive evaluation, we observe that even the most advanced visual
reasoning models (e.g., Gemini-2.5-pro and o4-mini) achieve sub-60\% accuracy
on our benchmark. These results reveal fundamental challenges in current large
language models' visual understanding capabilities, particularly in: (i)
establishing rigorous coupling between diagram interpretation and physics
reasoning, and (ii) overcoming their persistent reliance on textual cues as
cognitive shortcuts.
68367c93f6cadba33fdfb17c
Kun Xiang
68367c93f6cadba33fdfb17d
Heng Li
claimed_verified
2025-05-28T08:57:31.603Z
67604fe49dec814e4b7b772e
https://cdn-avatars.huggingface.co/v1/production/uploads/67604fe49dec814e4b7b772e/XQNDrAIj4NIP-ZZsRQg_2.jpeg
HengLi
user
HengLi29
68367c93f6cadba33fdfb17e
Terry Jingchen Zhang
68367c93f6cadba33fdfb17f
Yinya Huang
claimed_verified
2025-05-28T08:57:28.968Z
6628c1d30ccfcdcc321fc624
/avatars/016674fdb50219847e20aa1130a9e882.svg
Yinya Eleanor Huang
user
yinyahuang
68367c93f6cadba33fdfb180
Zirong Liu
68367c93f6cadba33fdfb181
Peixin Qu
68367c93f6cadba33fdfb182
Jixi He
68367c93f6cadba33fdfb183
Jiaqi Chen
68367c93f6cadba33fdfb184
Yu-Jie Yuan
68367c93f6cadba33fdfb185
Jianhua Han
68367c93f6cadba33fdfb186
Hang Xu
68367c93f6cadba33fdfb187
Hanhui Li
68367c93f6cadba33fdfb188
Mrinmaya Sachan
68367c93f6cadba33fdfb189
Xiaodan Liang
SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics
  Reasoning
We present SeePhys, a large-scale multimodal benchmark for LLM reasoning
grounded in physics questions ranging from middle school to PhD qualifying
exams. The benchmark covers 7 fundamental domains spanning the physics
discipline, incorporating 21 categories of highly heterogeneous diagrams. In
contrast to prior works where visual elements mainly serve auxiliary purposes,
our benchmark features a substantial proportion of vision-essential problems
(75\%) that mandate visual information extraction for correct solutions.
Through extensive evaluation, we observe that even the most advanced visual
reasoning models (e.g., Gemini-2.5-pro and o4-mini) achieve sub-60\% accuracy
on our benchmark. These results reveal fundamental challenges in current large
language models' visual understanding capabilities, particularly in: (i)
establishing rigorous coupling between diagram interpretation and physics
reasoning, and (ii) overcoming their persistent reliance on textual cues as
cognitive shortcuts.
SeePhys, a multimodal benchmark, highlights challenges in LLMs' visual reasoning and physics-grounded problem-solving capabilities, especially in interpreting diagrams and reducing reliance on textual cues.
LLM reasoning multimodal benchmark vision-essential problems visual information extraction visual reasoning models diagram interpretation physics reasoning cognitive shortcuts