Are large language models superhuman chemists?
Large language models (LLMs) have gained widespread interest due to their
ability to process human language and perform tasks on which they have not been
explicitly trained. This is relevant for the chemical sciences, which face the
problem of small and diverse datasets that are frequently in the form of text.
LLMs have shown promise in addressing these issues and are increasingly being
harnessed to predict chemical properties, optimize reactions, and even design
and conduct experiments autonomously. However, we still have only a very
limited systematic understanding of the chemical reasoning capabilities of
LLMs, which would be required to improve models and mitigate potential harms.
Here, we introduce "ChemBench," an automated framework designed to rigorously
evaluate the chemical knowledge and reasoning abilities of state-of-the-art
LLMs against the expertise of human chemists. We curated more than 7,000
question-answer pairs for a wide array of subfields of the chemical sciences,
evaluated leading open and closed-source LLMs, and found that the best models
outperformed the best human chemists in our study on average. The models,
however, struggle with some chemical reasoning tasks that are easy for human
experts and provide overconfident, misleading predictions, such as about
chemicals' safety profiles. These findings underscore the dual reality that,
although LLMs demonstrate remarkable proficiency in chemical tasks, further
research is critical to enhancing their safety and utility in chemical
sciences. Our findings also indicate a need for adaptations to chemistry
curricula and highlight the importance of continuing to develop evaluation
frameworks to improve safe and useful LLMs.
660cbe6099760c2ef3b1a9eb
Adrian Mirza
claimed_verified
2024-05-13T07:49:29.699Z
64dde64b8276275de53faf79
/avatars/32b438efaca7b83b8923b5ec34880faf.svg
Adrian Mirza
user
AdrianM0
660cbe6099760c2ef3b1a9ec
Nawaf Alampara
claimed_verified
2024-06-03T18:26:34.228Z
651dbb2f56b71e7dbc9a95c4
https://cdn-avatars.huggingface.co/v1/production/uploads/651dbb2f56b71e7dbc9a95c4/hD4OJnyEruOJF_2KaiF1D.png
Nawaf Alampara
user
n0w0f
660cbe6099760c2ef3b1a9ed
Sreekanth Kunchapu
660cbe6099760c2ef3b1a9ee
Benedict Emoekabu
660cbe6099760c2ef3b1a9ef
Aswanth Krishnan
660cbe6099760c2ef3b1a9f0
Mara Wilhelmi
660cbe6099760c2ef3b1a9f1
Macjonathan Okereke
660cbe6099760c2ef3b1a9f2
Juliane Eberhardt
660cbe6099760c2ef3b1a9f3
Amir Mohammad Elahi
660cbe6099760c2ef3b1a9f4
Maximilian Greiner
660cbe6099760c2ef3b1a9f5
Caroline T. Holick
660cbe6099760c2ef3b1a9f6
Tanya Gupta
660cbe6099760c2ef3b1a9f7
Mehrdad Asgari
660cbe6099760c2ef3b1a9f8
Christina Glaubitz
660cbe6099760c2ef3b1a9f9
Lea C. Klepsch
660cbe6099760c2ef3b1a9fa
Yannik Köster
660cbe6099760c2ef3b1a9fb
Jakob Meyer
660cbe6099760c2ef3b1a9fc
Santiago Miret
660cbe6099760c2ef3b1a9fd
Tim Hoffmann
660cbe6099760c2ef3b1a9fe
Fabian Alexander Kreth
660cbe6099760c2ef3b1a9ff
Michael Ringleb
660cbe6099760c2ef3b1aa00
Nicole Roesner
660cbe6099760c2ef3b1aa01
Ulrich S. Schubert
660cbe6099760c2ef3b1aa02
Leanne M. Stafast
660cbe6099760c2ef3b1aa03
Dinga Wonanke
660cbe6099760c2ef3b1aa04
Michael Pieler
claimed_verified
2024-04-24T11:12:17.385Z
6037f54f770949ef34f12d99
https://cdn-avatars.huggingface.co/v1/production/uploads/1614280013027-noauth.jpeg
Michael Pieler
user
MicPie
660cbe6099760c2ef3b1aa05
Philippe Schwaller
660cbe6099760c2ef3b1aa06
Kevin Maik Jablonka
Are large language models superhuman chemists?
Large language models (LLMs) have gained widespread interest due to their
ability to process human language and perform tasks on which they have not been
explicitly trained. This is relevant for the chemical sciences, which face the
problem of small and diverse datasets that are frequently in the form of text.
LLMs have shown promise in addressing these issues and are increasingly being
harnessed to predict chemical properties, optimize reactions, and even design
and conduct experiments autonomously. However, we still have only a very
limited systematic understanding of the chemical reasoning capabilities of
LLMs, which would be required to improve models and mitigate potential harms.
Here, we introduce "ChemBench," an automated framework designed to rigorously
evaluate the chemical knowledge and reasoning abilities of state-of-the-art
LLMs against the expertise of human chemists. We curated more than 7,000
question-answer pairs for a wide array of subfields of the chemical sciences,
evaluated leading open and closed-source LLMs, and found that the best models
outperformed the best human chemists in our study on average. The models,
however, struggle with some chemical reasoning tasks that are easy for human
experts and provide overconfident, misleading predictions, such as about
chemicals' safety profiles. These findings underscore the dual reality that,
although LLMs demonstrate remarkable proficiency in chemical tasks, further
research is critical to enhancing their safety and utility in chemical
sciences. Our findings also indicate a need for adaptations to chemistry
curricula and highlight the importance of continuing to develop evaluation
frameworks to improve safe and useful LLMs.
ChemBench, an automated evaluation framework, assesses the chemical reasoning of large language models against human experts, revealing their proficiency and shortcomings in specific tasks.
large language models LLMs chemical sciences datasets text chemical properties reactions experiments chemical reasoning ChemBench question-answer pairs subfields chemical knowledge human chemists safety profiles adaptations curricula evaluation frameworks