PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning
Large language models demonstrate remarkable capabilities across various
domains, especially mathematics and logic reasoning. However, current
evaluations overlook physics-based reasoning - a complex task requiring physics
theorems and constraints. We present PhysReason, a 1,200-problem benchmark
comprising knowledge-based (25%) and reasoning-based (75%) problems, where the
latter are divided into three difficulty levels (easy, medium, hard). Notably,
problems require an average of 8.1 solution steps, with hard requiring 15.6,
reflecting the complexity of physics-based reasoning. We propose the Physics
Solution Auto Scoring Framework, incorporating efficient answer-level and
comprehensive step-level evaluations. Top-performing models like Deepseek-R1,
Gemini-2.0-Flash-Thinking, and o3-mini-high achieve less than 60% on
answer-level evaluation, with performance dropping from knowledge questions
(75.11%) to hard problems (31.95%). Through step-level evaluation, we
identified four key bottlenecks: Physics Theorem Application, Physics Process
Understanding, Calculation, and Physics Condition Analysis. These findings
position PhysReason as a novel and comprehensive benchmark for evaluating
physics-based reasoning capabilities in large language models. Our code and
data will be published at https:/dxzxy12138.github.io/PhysReason.
67b44a6888813676da9f8239
Xinyu Zhang
67b44a6888813676da9f823a
Yuxuan Dong
67b44a6888813676da9f823b
Yanrui Wu
67b44a6888813676da9f823c
Jiaxing Huang
67b44a6888813676da9f823d
Chengyou Jia
claimed_verified
2025-02-18T09:30:47.313Z
6602548a68d519ed324b47c5
/avatars/5ab411f87440cc2a98c7a1c6a3ed5548.svg
ChengyouJia
user
ChengyouJia
67b44a6888813676da9f823e
Basura Fernando
67b44a6888813676da9f823f
Mike Zheng Shou
67b44a6888813676da9f8240
Lingling Zhang
67b44a6888813676da9f8241
Jun Liu
PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning
Large language models demonstrate remarkable capabilities across various
domains, especially mathematics and logic reasoning. However, current
evaluations overlook physics-based reasoning - a complex task requiring physics
theorems and constraints. We present PhysReason, a 1,200-problem benchmark
comprising knowledge-based (25%) and reasoning-based (75%) problems, where the
latter are divided into three difficulty levels (easy, medium, hard). Notably,
problems require an average of 8.1 solution steps, with hard requiring 15.6,
reflecting the complexity of physics-based reasoning. We propose the Physics
Solution Auto Scoring Framework, incorporating efficient answer-level and
comprehensive step-level evaluations. Top-performing models like Deepseek-R1,
Gemini-2.0-Flash-Thinking, and o3-mini-high achieve less than 60% on
answer-level evaluation, with performance dropping from knowledge questions
(75.11%) to hard problems (31.95%). Through step-level evaluation, we
identified four key bottlenecks: Physics Theorem Application, Physics Process
Understanding, Calculation, and Physics Condition Analysis. These findings
position PhysReason as a novel and comprehensive benchmark for evaluating
physics-based reasoning capabilities in large language models. Our code and
data will be published at https:/dxzxy12138.github.io/PhysReason.
PhysReason is a 1,200-problem benchmark designed to evaluate large language models' physics-based reasoning capabilities, identifying critical bottlenecks such as physics theorem application and calculation.
large language models physics-based reasoning PhysReason knowledge-based problems reasoning-based problems Physics Solution Auto Scoring Framework Deepseek-R1 Gemini-2.0-Flash-Thinking o3-mini-high physics theorem application physics process understanding physics condition analysis