CMPhysBench: A Benchmark for Evaluating Large Language Models in
  Condensed Matter Physics
We introduce CMPhysBench, designed to assess the proficiency of Large
Language Models (LLMs) in Condensed Matter Physics, as a novel Benchmark.
CMPhysBench is composed of more than 520 graduate-level meticulously curated
questions covering both representative subfields and foundational theoretical
frameworks of condensed matter physics, such as magnetism, superconductivity,
strongly correlated systems, etc. To ensure a deep understanding of the
problem-solving process,we focus exclusively on calculation problems, requiring
LLMs to independently generate comprehensive solutions. Meanwhile, leveraging
tree-based representations of expressions, we introduce the Scalable Expression
Edit Distance (SEED) score, which provides fine-grained (non-binary) partial
credit and yields a more accurate assessment of similarity between prediction
and ground-truth. Our results show that even the best models, Grok-4, reach
only 36 average SEED score and 28% accuracy on CMPhysBench, underscoring a
significant capability gap, especially for this practical and frontier domain
relative to traditional physics. The code anddataset are publicly available at
https://github.com/CMPhysBench/CMPhysBench.
68ae7da2364411bea07df7d1
Weida Wang
claimed_verified
2025-08-27T07:10:33.081Z
661b9d96c153e4a0a25adc3e
https://cdn-avatars.huggingface.co/v1/production/uploads/661b9d96c153e4a0a25adc3e/VRt7kCQ0KdJp-lhPLOajO.jpeg
Weida Wang
user
weidawang
68ae7da2364411bea07df7d2
Dongchen Huang
68ae7da2364411bea07df7d3
Jiatong Li
68ae7da2364411bea07df7d4
Tengchao Yang
68ae7da2364411bea07df7d5
Ziyang Zheng
68ae7da2364411bea07df7d6
Di Zhang
claimed_verified
2025-08-27T07:10:30.932Z
64bce15bafd1e46c5504ad38
https://cdn-avatars.huggingface.co/v1/production/uploads/64bce15bafd1e46c5504ad38/vkEjiu-mIagKlrXzDH75o.png
Di Zhang
user
di-zhang-fdu
68ae7da2364411bea07df7d7
Dong Han
68ae7da2364411bea07df7d8
Benteng Chen
68ae7da2364411bea07df7d9
Binzhao Luo
68ae7da2364411bea07df7da
Zhiyu Liu
68ae7da2364411bea07df7db
Kunling Liu
68ae7da2364411bea07df7dc
Zhiyuan Gao
68ae7da2364411bea07df7dd
Shiqi Geng
68ae7da2364411bea07df7de
Wei Ma
claimed_verified
2025-08-27T07:10:28.667Z
689d58bb635465616d75d302
/avatars/68797cb28dc87478c791d5152e62c52a.svg
Wei Ma
user
BoringMarsh
68ae7da2364411bea07df7df
Jiaming Su
68ae7da2364411bea07df7e0
Xin Li
68ae7da2364411bea07df7e1
Shuchen Pu
claimed_verified
2025-08-27T10:02:16.074Z
67348828c74a3af1aef4d3a2
https://cdn-avatars.huggingface.co/v1/production/uploads/67348828c74a3af1aef4d3a2/ToZ56JIM3ccAXIBNG4mx_.jpeg
pushuchen
user
komusama0930
68ae7da2364411bea07df7e2
Yuhan Shui
68ae7da2364411bea07df7e3
Qianjia Cheng
68ae7da2364411bea07df7e4
Zhihao Dou
68ae7da2364411bea07df7e5
Dongfei Cui
68ae7da2364411bea07df7e6
Changyong He
68ae7da2364411bea07df7e7
Jin Zeng
68ae7da2364411bea07df7e8
Zeke Xie
68ae7da2364411bea07df7e9
Mao Su
68ae7da2364411bea07df7ea
Dongzhan Zhou
68ae7da2364411bea07df7eb
Yuqiang Li
68ae7da2364411bea07df7ec
Wanli Ouyang
68ae7da2364411bea07df7ed
Yunqi Cai
68ae7da2364411bea07df7ee
Xi Dai
68ae7da2364411bea07df7ef
Shufei Zhang
68ae7da2364411bea07df7f0
Lei Bai
68ae7da2364411bea07df7f1
Jinguang Cheng
68ae7da2364411bea07df7f2
Zhong Fang
68ae7da2364411bea07df7f3
Hongming Weng
CMPhysBench: A Benchmark for Evaluating Large Language Models in
  Condensed Matter Physics
We introduce CMPhysBench, designed to assess the proficiency of Large
Language Models (LLMs) in Condensed Matter Physics, as a novel Benchmark.
CMPhysBench is composed of more than 520 graduate-level meticulously curated
questions covering both representative subfields and foundational theoretical
frameworks of condensed matter physics, such as magnetism, superconductivity,
strongly correlated systems, etc. To ensure a deep understanding of the
problem-solving process,we focus exclusively on calculation problems, requiring
LLMs to independently generate comprehensive solutions. Meanwhile, leveraging
tree-based representations of expressions, we introduce the Scalable Expression
Edit Distance (SEED) score, which provides fine-grained (non-binary) partial
credit and yields a more accurate assessment of similarity between prediction
and ground-truth. Our results show that even the best models, Grok-4, reach
only 36 average SEED score and 28% accuracy on CMPhysBench, underscoring a
significant capability gap, especially for this practical and frontier domain
relative to traditional physics. The code anddataset are publicly available at
https://github.com/CMPhysBench/CMPhysBench.
CMPhysBench evaluates LLMs in condensed matter physics using calculation problems and a new SEED score for partial credit assessment, revealing significant capability gaps.
Large Language Models (LLMs) Condensed Matter Physics Scalable Expression Edit Distance (SEED) tree-based representations Grok-4