LAB-Bench: Measuring Capabilities of Language Models for Biology
  Research
There is widespread optimism that frontier Large Language Models (LLMs) and
LLM-augmented systems have the potential to rapidly accelerate scientific
discovery across disciplines. Today, many benchmarks exist to measure LLM
knowledge and reasoning on textbook-style science questions, but few if any
benchmarks are designed to evaluate language model performance on practical
tasks required for scientific research, such as literature search, protocol
planning, and data analysis. As a step toward building such benchmarks, we
introduce the Language Agent Biology Benchmark (LAB-Bench), a broad dataset of
over 2,400 multiple choice questions for evaluating AI systems on a range of
practical biology research capabilities, including recall and reasoning over
literature, interpretation of figures, access and navigation of databases, and
comprehension and manipulation of DNA and protein sequences. Importantly, in
contrast to previous scientific benchmarks, we expect that an AI system that
can achieve consistently high scores on the more difficult LAB-Bench tasks
would serve as a useful assistant for researchers in areas such as literature
search and molecular cloning. As an initial assessment of the emergent
scientific task capabilities of frontier language models, we measure
performance of several against our benchmark and report results compared to
human expert biology researchers. We will continue to update and expand
LAB-Bench over time, and expect it to serve as a useful tool in the development
of automated research systems going forward. A public subset of LAB-Bench is
available for use at the following URL:
https://huggingface.co/datasets/futurehouse/lab-bench
6695cb23d4ca2767b9d5a15f
Jon M. Laurent
6695cb23d4ca2767b9d5a160
Joseph D. Janizek
6695cb23d4ca2767b9d5a161
Michael Ruzo
6695cb23d4ca2767b9d5a162
Michaela M. Hinks
6695cb23d4ca2767b9d5a163
Michael J. Hammerling
6695cb23d4ca2767b9d5a164
Siddharth Narayanan
6695cb23d4ca2767b9d5a165
Manvitha Ponnapati
6695cb23d4ca2767b9d5a166
Andrew D. White
6695cb23d4ca2767b9d5a167
Samuel G. Rodriques
LAB-Bench: Measuring Capabilities of Language Models for Biology
  Research
There is widespread optimism that frontier Large Language Models (LLMs) and
LLM-augmented systems have the potential to rapidly accelerate scientific
discovery across disciplines. Today, many benchmarks exist to measure LLM
knowledge and reasoning on textbook-style science questions, but few if any
benchmarks are designed to evaluate language model performance on practical
tasks required for scientific research, such as literature search, protocol
planning, and data analysis. As a step toward building such benchmarks, we
introduce the Language Agent Biology Benchmark (LAB-Bench), a broad dataset of
over 2,400 multiple choice questions for evaluating AI systems on a range of
practical biology research capabilities, including recall and reasoning over
literature, interpretation of figures, access and navigation of databases, and
comprehension and manipulation of DNA and protein sequences. Importantly, in
contrast to previous scientific benchmarks, we expect that an AI system that
can achieve consistently high scores on the more difficult LAB-Bench tasks
would serve as a useful assistant for researchers in areas such as literature
search and molecular cloning. As an initial assessment of the emergent
scientific task capabilities of frontier language models, we measure
performance of several against our benchmark and report results compared to
human expert biology researchers. We will continue to update and expand
LAB-Bench over time, and expect it to serve as a useful tool in the development
of automated research systems going forward. A public subset of LAB-Bench is
available for use at the following URL:
https://huggingface.co/datasets/futurehouse/lab-bench
LAB-Bench, a new dataset, evaluates AI systems on practical biology research tasks, providing a benchmark for assessing scientific reasoning and practical capabilities beyond textbook-style questions.
Large Language Models LLM-augmented systems literature search protocol planning data analysis multiple choice questions recall reasoning over literature interpretation of figures database access and navigation DNA and protein sequence comprehension automated research systems