BoxingGym: Benchmarking Progress in Automated Experimental Design and
  Model Discovery
Understanding the world and explaining it with scientific theories is a
central aspiration of artificial intelligence research. Proposing theories,
designing experiments to test them, and then revising them based on data are
fundamental to scientific discovery. Despite the significant promise of
LLM-based scientific agents, no benchmarks systematically test LLM's ability to
propose scientific models, collect experimental data, and revise them in light
of new data. We introduce BoxingGym, a benchmark with 10 environments for
systematically evaluating both experimental design (e.g. collecting data to
test a scientific theory) and model discovery (e.g. proposing and revising
scientific theories). To enable tractable and quantitative evaluation, we
implement each environment as a generative probabilistic model with which a
scientific agent can run interactive experiments. These probabilistic models
are drawn from various real-world scientific domains ranging from psychology to
ecology. To quantitatively evaluate a scientific agent's ability to collect
informative experimental data, we compute the expected information gain (EIG),
an information-theoretic quantity which measures how much an experiment reduces
uncertainty about the parameters of a generative model. A good scientific
theory is a concise and predictive explanation. Therefore, to quantitatively
evaluate model discovery, we ask a scientific agent to explain their model and
then assess whether this explanation enables another scientific agent to make
reliable predictions about this environment. In addition to this
explanation-based evaluation, we compute standard model evaluation metrics such
as prediction errors. We find that current LLMs, such as GPT-4o, struggle with
both experimental design and model discovery. We find that augmenting the
LLM-based agent with an explicit statistical model does not reliably improve
these results.
677b9534c57d565bd3535ff2
Kanishk Gandhi
extracted_confirmed
2025-01-06T08:33:22.650Z
63e6a880f2e9a8f22c5a1630
/avatars/53b57690fe052ce6882bbfc87b11567c.svg
Kanishk Gandhi
user
obiwan96
677b9534c57d565bd3535ff3
Michael Y. Li
admin_assigned
2025-01-06T08:36:16.321Z
667357d37376c7227771c14c
/avatars/77253b6f8b1eb97f9ab09097e83436ed.svg
Michael Y Li
user
michaelyli
677b9534c57d565bd3535ff4
Lyle Goodyear
677b9534c57d565bd3535ff5
Louise Li
677b9534c57d565bd3535ff6
Aditi Bhaskar
677b9534c57d565bd3535ff7
Mohammed Zaman
677b9534c57d565bd3535ff8
Noah D. Goodman
admin_assigned
2025-01-06T08:36:39.125Z
67321274c1f20c742bcf7a8d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ltcQhre6eDRVzn6Vbbyhu.png
Noah D. Goodman
user
ngoodman
BoxingGym: Benchmarking Progress in Automated Experimental Design and
  Model Discovery
Understanding the world and explaining it with scientific theories is a
central aspiration of artificial intelligence research. Proposing theories,
designing experiments to test them, and then revising them based on data are
fundamental to scientific discovery. Despite the significant promise of
LLM-based scientific agents, no benchmarks systematically test LLM's ability to
propose scientific models, collect experimental data, and revise them in light
of new data. We introduce BoxingGym, a benchmark with 10 environments for
systematically evaluating both experimental design (e.g. collecting data to
test a scientific theory) and model discovery (e.g. proposing and revising
scientific theories). To enable tractable and quantitative evaluation, we
implement each environment as a generative probabilistic model with which a
scientific agent can run interactive experiments. These probabilistic models
are drawn from various real-world scientific domains ranging from psychology to
ecology. To quantitatively evaluate a scientific agent's ability to collect
informative experimental data, we compute the expected information gain (EIG),
an information-theoretic quantity which measures how much an experiment reduces
uncertainty about the parameters of a generative model. A good scientific
theory is a concise and predictive explanation. Therefore, to quantitatively
evaluate model discovery, we ask a scientific agent to explain their model and
then assess whether this explanation enables another scientific agent to make
reliable predictions about this environment. In addition to this
explanation-based evaluation, we compute standard model evaluation metrics such
as prediction errors. We find that current LLMs, such as GPT-4o, struggle with
both experimental design and model discovery. We find that augmenting the
LLM-based agent with an explicit statistical model does not reliably improve
these results.
BoxingGym benchmark evaluates LLMs' ability to design experiments and propose scientific models across various domains using generative probabilistic models and information-theoretic metrics.
LLM-based scientific agents BoxingGym experimental design model discovery generative probabilistic models expected information gain (EIG) information-theoretic quantity scientific agent explanation-based evaluation model evaluation metrics prediction errors