PHYBench: Holistic Evaluation of Physical Perception and Reasoning in
  Large Language Models
We introduce PHYBench, a novel, high-quality benchmark designed for
evaluating reasoning capabilities of large language models (LLMs) in physical
contexts. PHYBench consists of 500 meticulously curated physics problems based
on real-world physical scenarios, designed to assess the ability of models to
understand and reason about realistic physical processes. Covering mechanics,
electromagnetism, thermodynamics, optics, modern physics, and advanced physics,
the benchmark spans difficulty levels from high school exercises to
undergraduate problems and Physics Olympiad challenges. Additionally, we
propose the Expression Edit Distance (EED) Score, a novel evaluation metric
based on the edit distance between mathematical expressions, which effectively
captures differences in model reasoning processes and results beyond
traditional binary scoring methods. We evaluate various LLMs on PHYBench and
compare their performance with human experts. Our results reveal that even
state-of-the-art reasoning models significantly lag behind human experts,
highlighting their limitations and the need for improvement in complex physical
reasoning scenarios. Our benchmark results and dataset are publicly available
at https://phybench-official.github.io/phybench-demo/.
680a12f28ea8fd5e81332f3e
Shi Qiu
claimed_verified
2025-04-25T08:35:06.205Z
6644bb2c9bdbd85493074411
https://cdn-avatars.huggingface.co/v1/production/uploads/6644bb2c9bdbd85493074411/iug8dkP1zjID-kXzNxDjD.jpeg
SHI QIU
user
StarThomas1002
680a12f28ea8fd5e81332f3f
Shaoyang Guo
680a12f28ea8fd5e81332f40
Zhuo-Yang Song
680a12f28ea8fd5e81332f41
Yunbo Sun
680a12f28ea8fd5e81332f42
Zeyu Cai
680a12f28ea8fd5e81332f43
Jiashen Wei
680a12f28ea8fd5e81332f44
Tianyu Luo
claimed_verified
2025-04-25T08:35:11.506Z
68079e7d34496fa3eec1adeb
https://cdn-avatars.huggingface.co/v1/production/uploads/68079e7d34496fa3eec1adeb/qSpUFt6nwb8cu3GX9cd5y.jpeg
Tianyu Luo
user
CaCtusLuo
680a12f28ea8fd5e81332f45
Yixuan Yin
680a12f28ea8fd5e81332f46
Haoxu Zhang
680a12f28ea8fd5e81332f47
Yi Hu
680a12f28ea8fd5e81332f48
Chenyang Wang
680a12f28ea8fd5e81332f49
Chencheng Tang
680a12f28ea8fd5e81332f4a
Haoling Chang
680a12f28ea8fd5e81332f4b
Qi Liu
680a12f28ea8fd5e81332f4c
Ziheng Zhou
680a12f28ea8fd5e81332f4d
Tianyu Zhang
680a12f28ea8fd5e81332f4e
Jingtian Zhang
680a12f28ea8fd5e81332f4f
Zhangyi Liu
680a12f28ea8fd5e81332f50
Minghao Li
680a12f28ea8fd5e81332f51
Yuku Zhang
680a12f28ea8fd5e81332f52
Boxuan Jing
680a12f28ea8fd5e81332f53
Xianqi Yin
680a12f28ea8fd5e81332f54
Yutong Ren
680a12f28ea8fd5e81332f55
Zizhuo Fu
claimed_verified
2025-04-25T08:35:09.065Z
67c7b67cbc99f666def4e1de
/avatars/818967a977070b32a4f2037bd377a92d.svg
zZzZzZ
user
zZzZzZ9999999
680a12f28ea8fd5e81332f56
Weike Wang
680a12f28ea8fd5e81332f57
Xudong Tian
680a12f28ea8fd5e81332f58
Anqi Lv
680a12f28ea8fd5e81332f59
Laifu Man
680a12f28ea8fd5e81332f5a
Jianxiang Li
680a12f28ea8fd5e81332f5b
Feiyu Tao
680a12f28ea8fd5e81332f5c
Qihua Sun
680a12f28ea8fd5e81332f5d
Zhou Liang
680a12f28ea8fd5e81332f5e
Yushu Mu
680a12f28ea8fd5e81332f5f
Zhongxuan Li
680a12f28ea8fd5e81332f60
Jing-Jun Zhang
680a12f28ea8fd5e81332f61
Shutao Zhang
680a12f28ea8fd5e81332f62
Xiaotian Li
680a12f28ea8fd5e81332f63
Xingqi Xia
680a12f28ea8fd5e81332f64
Jiawei Lin
680a12f28ea8fd5e81332f65
Zheyu Shen
680a12f28ea8fd5e81332f66
Jiahang Chen
680a12f28ea8fd5e81332f67
Qiuhao Xiong
680a12f28ea8fd5e81332f68
Binran Wang
680a12f28ea8fd5e81332f69
Fengyuan Wang
680a12f28ea8fd5e81332f6a
Ziyang Ni
680a12f28ea8fd5e81332f6b
Bohan Zhang
680a12f28ea8fd5e81332f6c
Fan Cui
680a12f28ea8fd5e81332f6d
Changkun Shao
680a12f28ea8fd5e81332f6e
Qing-Hong Cao
680a12f28ea8fd5e81332f6f
Ming-xing Luo
680a12f28ea8fd5e81332f70
Muhan Zhang
680a12f28ea8fd5e81332f71
Hua Xing Zhu
PHYBench: Holistic Evaluation of Physical Perception and Reasoning in
  Large Language Models
We introduce PHYBench, a novel, high-quality benchmark designed for
evaluating reasoning capabilities of large language models (LLMs) in physical
contexts. PHYBench consists of 500 meticulously curated physics problems based
on real-world physical scenarios, designed to assess the ability of models to
understand and reason about realistic physical processes. Covering mechanics,
electromagnetism, thermodynamics, optics, modern physics, and advanced physics,
the benchmark spans difficulty levels from high school exercises to
undergraduate problems and Physics Olympiad challenges. Additionally, we
propose the Expression Edit Distance (EED) Score, a novel evaluation metric
based on the edit distance between mathematical expressions, which effectively
captures differences in model reasoning processes and results beyond
traditional binary scoring methods. We evaluate various LLMs on PHYBench and
compare their performance with human experts. Our results reveal that even
state-of-the-art reasoning models significantly lag behind human experts,
highlighting their limitations and the need for improvement in complex physical
reasoning scenarios. Our benchmark results and dataset are publicly available
at https://phybench-official.github.io/phybench-demo/.
A new benchmark evaluates large language models on physics problems with a novel Expression Edit Distance metric, revealing gaps compared to human experts.
PHYBench large language models LLMs physics problems mechanics electromagnetism thermodynamics optics modern physics advanced physics Expression Edit Distance EED Score