PhysGym: Benchmarking LLMs in Interactive Physics Discovery with
  Controlled Priors
Evaluating the scientific discovery capabilities of large language model
based agents, particularly how they cope with varying environmental complexity
and utilize prior knowledge, requires specialized benchmarks currently lacking
in the landscape. To address this gap, we introduce PhysGym, a novel benchmark
suite and simulation platform for rigorously assessing LLM-based scientific
reasoning in interactive physics environments. PhysGym's primary contribution
lies in its sophisticated control over the level of prior knowledge provided to
the agent. This allows researchers to dissect agent performance along axes
including the complexity of the problem and the prior knowledge levels. The
benchmark comprises a suite of interactive simulations, where agents must
actively probe environments, gather data sequentially under constraints and
formulate hypotheses about underlying physical laws. PhysGym provides
standardized evaluation protocols and metrics for assessing hypothesis accuracy
and model fidelity. We demonstrate the benchmark's utility by presenting
results from baseline LLMs, showcasing its ability to differentiate
capabilities based on varying priors and task complexity.
687f3f6133947f780d9b4b8d
Yimeng Chen
claimed_verified
2025-07-22T13:22:30.313Z
6836f97bed0b13d776f1006a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/-h4S4OJYJG02PFkYXKPYX.png
Yimeng Chen
user
YimengChen
687f3f6133947f780d9b4b8e
Piotr Piȩkos
687f3f6133947f780d9b4b8f
Mateusz Ostaszewski
687f3f6133947f780d9b4b90
Firas Laakom
687f3f6133947f780d9b4b91
Jürgen Schmidhuber
PhysGym: Benchmarking LLMs in Interactive Physics Discovery with
  Controlled Priors
Evaluating the scientific discovery capabilities of large language model
based agents, particularly how they cope with varying environmental complexity
and utilize prior knowledge, requires specialized benchmarks currently lacking
in the landscape. To address this gap, we introduce PhysGym, a novel benchmark
suite and simulation platform for rigorously assessing LLM-based scientific
reasoning in interactive physics environments. PhysGym's primary contribution
lies in its sophisticated control over the level of prior knowledge provided to
the agent. This allows researchers to dissect agent performance along axes
including the complexity of the problem and the prior knowledge levels. The
benchmark comprises a suite of interactive simulations, where agents must
actively probe environments, gather data sequentially under constraints and
formulate hypotheses about underlying physical laws. PhysGym provides
standardized evaluation protocols and metrics for assessing hypothesis accuracy
and model fidelity. We demonstrate the benchmark's utility by presenting
results from baseline LLMs, showcasing its ability to differentiate
capabilities based on varying priors and task complexity.
PhysGym is a benchmark suite and simulation platform for evaluating large language model-based scientific reasoning in interactive physics environments, allowing researchers to assess performance across different levels of prior knowledge and problem complexity.
large language model scientific discovery environmental complexity prior knowledge benchmark suite simulation platform interactive physics environments hypothesis accuracy model fidelity