Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics
  Research Benchmark
While large language models (LLMs) with reasoning capabilities are
progressing rapidly on high-school math competitions and coding, can they
reason effectively through complex, open-ended challenges found in frontier
physics research? And crucially, what kinds of reasoning tasks do physicists
want LLMs to assist with? To address these questions, we present the CritPt
(Complex Research using Integrated Thinking - Physics Test, pronounced
"critical point"), the first benchmark designed to test LLMs on unpublished,
research-level reasoning tasks that broadly covers modern physics research
areas, including condensed matter, quantum physics, atomic, molecular & optical
physics, astrophysics, high energy physics, mathematical physics, statistical
physics, nuclear physics, nonlinear dynamics, fluid dynamics and biophysics.
CritPt consists of 71 composite research challenges designed to simulate
full-scale research projects at the entry level, which are also decomposed to
190 simpler checkpoint tasks for more fine-grained insights. All problems are
newly created by 50+ active physics researchers based on their own research.
Every problem is hand-curated to admit a guess-resistant and machine-verifiable
answer and is evaluated by an automated grading pipeline heavily customized for
advanced physics-specific output formats. We find that while current
state-of-the-art LLMs show early promise on isolated checkpoints, they remain
far from being able to reliably solve full research-scale challenges: the best
average accuracy among base models is only 4.0% , achieved by GPT-5 (high),
moderately rising to around 10% when equipped with coding tools. Through the
realistic yet standardized evaluation offered by CritPt, we highlight a large
disconnect between current model capabilities and realistic physics research
demands, offering a foundation to guide the development of scientifically
grounded AI tools.
68dc94be4159d1f2418f9b01
Minhui Zhu
68dc94be4159d1f2418f9b02
Minyang Tian
68dc94be4159d1f2418f9b03
Xiaocheng Yang
68dc94be4159d1f2418f9b04
Tianci Zhou
68dc94be4159d1f2418f9b05
Penghao Zhu
68dc94be4159d1f2418f9b06
Eli Chertkov
68dc94be4159d1f2418f9b07
Shengyan Liu
68dc94be4159d1f2418f9b08
Yufeng Du
68dc94be4159d1f2418f9b09
Lifan Yuan
68dc94be4159d1f2418f9b0a
Ziming Ji
68dc94be4159d1f2418f9b0b
Indranil Das
68dc94be4159d1f2418f9b0c
Junyi Cao
68dc94be4159d1f2418f9b0d
Yufeng Du
68dc94be4159d1f2418f9b0e
Jinchen He
68dc94be4159d1f2418f9b0f
Yifan Su
68dc94be4159d1f2418f9b10
Jiabin Yu
68dc94be4159d1f2418f9b11
Yikun Jiang
68dc94be4159d1f2418f9b12
Yujie Zhang
68dc94be4159d1f2418f9b13
Chang Liu
68dc94be4159d1f2418f9b14
Ze-Min Huang
68dc94be4159d1f2418f9b15
Weizhen Jia
68dc94be4159d1f2418f9b16
Xinan Chen
68dc94be4159d1f2418f9b17
Peixue Wu
68dc94be4159d1f2418f9b18
Yunkai Wang
68dc94be4159d1f2418f9b19
Juntai Zhou
68dc94be4159d1f2418f9b1a
Yong Zhao
68dc94be4159d1f2418f9b1b
Farshid Jafarpour
68dc94be4159d1f2418f9b1c
Jessie Shelton
68dc94be4159d1f2418f9b1d
Aaron Young
68dc94be4159d1f2418f9b1e
John Bartolotta
68dc94be4159d1f2418f9b1f
Wenchao Xu
68dc94be4159d1f2418f9b20
Yue Sun
68dc94be4159d1f2418f9b21
Anjun Chu
68dc94be4159d1f2418f9b22
Victor Colussi
68dc94be4159d1f2418f9b23
Chris Akers
68dc94be4159d1f2418f9b24
Nathan Brooks
68dc94be4159d1f2418f9b25
Wenbo Fu
68dc94be4159d1f2418f9b26
Christopher Wilson
68dc94be4159d1f2418f9b27
Jinchao Zhao
68dc94be4159d1f2418f9b28
Marvin Qi
68dc94be4159d1f2418f9b29
Anqi Mu
68dc94be4159d1f2418f9b2a
Yubo Yang
68dc94be4159d1f2418f9b2b
Allen Zang
68dc94be4159d1f2418f9b2c
Yang Lyu
68dc94be4159d1f2418f9b2d
Peizhi Mai
68dc94be4159d1f2418f9b2e
Xuefei Guo
68dc94be4159d1f2418f9b2f
Luyu Gao
68dc94be4159d1f2418f9b30
Ze Yang
68dc94be4159d1f2418f9b31
Chi Xue
68dc94be4159d1f2418f9b32
Dmytro Bandak
68dc94be4159d1f2418f9b33
Yaïr Hein
68dc94be4159d1f2418f9b34
Yonatan Kahn
68dc94be4159d1f2418f9b35
Kevin Zhou
68dc94be4159d1f2418f9b36
John Drew Wilson Jarrod T. Reilly
68dc94be4159d1f2418f9b37
Di Luo
68dc94be4159d1f2418f9b38
Daniel Inafuku
68dc94be4159d1f2418f9b39
Hao Tong
68dc94be4159d1f2418f9b3a
Liang Yang
68dc94be4159d1f2418f9b3b
Ruixing Zhang
68dc94be4159d1f2418f9b3c
Xueying Wang
68dc94be4159d1f2418f9b3d
Ofir Press
68dc94be4159d1f2418f9b3e
Nicolas Chia
68dc94be4159d1f2418f9b3f
Eliu Huerta
68dc94be4159d1f2418f9b40
Hao Peng
Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics
  Research Benchmark
While large language models (LLMs) with reasoning capabilities are
progressing rapidly on high-school math competitions and coding, can they
reason effectively through complex, open-ended challenges found in frontier
physics research? And crucially, what kinds of reasoning tasks do physicists
want LLMs to assist with? To address these questions, we present the CritPt
(Complex Research using Integrated Thinking - Physics Test, pronounced
"critical point"), the first benchmark designed to test LLMs on unpublished,
research-level reasoning tasks that broadly covers modern physics research
areas, including condensed matter, quantum physics, atomic, molecular & optical
physics, astrophysics, high energy physics, mathematical physics, statistical
physics, nuclear physics, nonlinear dynamics, fluid dynamics and biophysics.
CritPt consists of 71 composite research challenges designed to simulate
full-scale research projects at the entry level, which are also decomposed to
190 simpler checkpoint tasks for more fine-grained insights. All problems are
newly created by 50+ active physics researchers based on their own research.
Every problem is hand-curated to admit a guess-resistant and machine-verifiable
answer and is evaluated by an automated grading pipeline heavily customized for
advanced physics-specific output formats. We find that while current
state-of-the-art LLMs show early promise on isolated checkpoints, they remain
far from being able to reliably solve full research-scale challenges: the best
average accuracy among base models is only 4.0% , achieved by GPT-5 (high),
moderately rising to around 10% when equipped with coding tools. Through the
realistic yet standardized evaluation offered by CritPt, we highlight a large
disconnect between current model capabilities and realistic physics research
demands, offering a foundation to guide the development of scientifically
grounded AI tools.
CritPt, a benchmark for evaluating LLMs on research-level physics tasks, reveals significant gaps between current model capabilities and the demands of physics research.
large language models reasoning capabilities high-school math competitions coding CritPt benchmark research-level reasoning tasks condensed matter quantum physics atomic molecular & optical physics astrophysics high energy physics mathematical physics statistical physics nuclear physics nonlinear dynamics fluid dynamics biophysics composite research challenges checkpoint tasks guess-resistant machine-verifiable automated grading pipeline advanced physics-specific output formats GPT-5