CURIE: Evaluating LLMs On Multitask Scientific Long Context
  Understanding and Reasoning
Scientific problem-solving involves synthesizing information while applying
expert knowledge. We introduce CURIE, a scientific long-Context
Understanding,Reasoning and Information Extraction benchmark to measure the
potential of Large Language Models (LLMs) in scientific problem-solving and
assisting scientists in realistic workflows. This benchmark introduces ten
challenging tasks with a total of 580 problems and solution pairs curated by
experts in six disciplines - materials science, condensed matter physics,
quantum computing, geospatial analysis, biodiversity, and proteins - covering
both experimental and theoretical work-flows in science. We evaluate a range of
closed and open LLMs on tasks in CURIE which requires domain expertise,
comprehension of long in-context information,and multi-step reasoning. While
Gemini Flash 2.0 and Claude-3 show consistent high comprehension across
domains, the popular GPT-4o and command-R+ fail dramatically on protein
sequencing tasks. With the best performance at 32% there is much room for
improvement for all models. We hope that insights gained from CURIE can guide
the future development of LLMs in sciences. Evaluation code and data are in
https://github.com/google/curie
67dc13a0b0451eaa2c5e298a
Hao Cui
67dc13a0b0451eaa2c5e298b
Zahra Shamsi
67dc13a0b0451eaa2c5e298c
Gowoon Cheon
67dc13a0b0451eaa2c5e298d
Xuejian Ma
67dc13a0b0451eaa2c5e298e
Shutong Li
admin_assigned
2025-03-20T13:31:25.035Z
668ec66fd76c5aee9ba82a57
/avatars/4ba953faaf05c483615d2ea1ae0b5572.svg
Shutong Li
user
LigeiaE
67dc13a0b0451eaa2c5e298f
Maria Tikhanovskaya
67dc13a0b0451eaa2c5e2990
Peter Norgaard
admin_assigned
2025-03-20T13:31:16.168Z
6751fd0c907b6ca2f6c71179
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/75-MmaXqwW_HFMDoHntkU.jpeg
Peter Norgaard
user
PNorgMagician
67dc13a0b0451eaa2c5e2991
Nayantara Mudur
67dc13a0b0451eaa2c5e2992
Martyna Plomecka
admin_assigned
2025-03-20T13:31:47.096Z
6251ade7183aa426692442a0
https://cdn-avatars.huggingface.co/v1/production/uploads/6251ade7183aa426692442a0/1nP_mQqgcrSPgV1Cf2nMP.jpeg
Martyna Plomecka
user
Martynaplomecka
67dc13a0b0451eaa2c5e2993
Paul Raccuglia
admin_assigned
2025-03-20T13:31:53.506Z
667c456320d68cc5814ebc04
/avatars/4cb3e893f36aac3442e46afae9d725c7.svg
Paul Raccuglia
user
praccu-google
67dc13a0b0451eaa2c5e2994
Yasaman Bahri
67dc13a0b0451eaa2c5e2995
Victor V. Albert
admin_assigned
2025-03-20T13:32:04.907Z
66a7bb1d5f395fc0c6913f77
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sDFjAHabkD_SEYACVD5YU.jpeg
Victor V. Albert
user
valbert4
67dc13a0b0451eaa2c5e2996
Pranesh Srinivasan
67dc13a0b0451eaa2c5e2997
Haining Pan
admin_assigned
2025-03-20T13:31:02.332Z
64664b4ecf550af36eb40363
/avatars/bf2b1ebbeac37f943578d5c93c834746.svg
Haining Pan
user
hainingpan
67dc13a0b0451eaa2c5e2998
Philippe Faist
67dc13a0b0451eaa2c5e2999
Brian Rohr
67dc13a0b0451eaa2c5e299a
Michael J. Statt
67dc13a0b0451eaa2c5e299b
Dan Morris
admin_assigned
2025-03-20T13:30:48.407Z
6386a8ed06858a85f595b533
/avatars/bdcd45db1dbc86d2f817bc682578cde3.svg
Dan Morris
user
danmorris427
67dc13a0b0451eaa2c5e299c
Drew Purves
67dc13a0b0451eaa2c5e299d
Elise Kleeman
67dc13a0b0451eaa2c5e299e
Ruth Alcantara
67dc13a0b0451eaa2c5e299f
Matthew Abraham
67dc13a0b0451eaa2c5e29a0
Muqthar Mohammad
67dc13a0b0451eaa2c5e29a1
Ean Phing VanLee
67dc13a0b0451eaa2c5e29a2
Chenfei Jiang
67dc13a0b0451eaa2c5e29a3
Elizabeth Dorfman
67dc13a0b0451eaa2c5e29a4
Eun-Ah Kim
67dc13a0b0451eaa2c5e29a5
Michael P Brenner
67dc13a0b0451eaa2c5e29a6
Viren Jain
67dc13a0b0451eaa2c5e29a7
Sameera Ponda
67dc13a0b0451eaa2c5e29a8
Subhashini Venugopalan
extracted_confirmed
2025-03-20T13:15:39.949Z
646125a5933afb0106a9dabb
/avatars/ccef359e4442f560a87fd66228e54c8b.svg
Subhashini
user
vsubhashini
CURIE: Evaluating LLMs On Multitask Scientific Long Context
  Understanding and Reasoning
Scientific problem-solving involves synthesizing information while applying
expert knowledge. We introduce CURIE, a scientific long-Context
Understanding,Reasoning and Information Extraction benchmark to measure the
potential of Large Language Models (LLMs) in scientific problem-solving and
assisting scientists in realistic workflows. This benchmark introduces ten
challenging tasks with a total of 580 problems and solution pairs curated by
experts in six disciplines - materials science, condensed matter physics,
quantum computing, geospatial analysis, biodiversity, and proteins - covering
both experimental and theoretical work-flows in science. We evaluate a range of
closed and open LLMs on tasks in CURIE which requires domain expertise,
comprehension of long in-context information,and multi-step reasoning. While
Gemini Flash 2.0 and Claude-3 show consistent high comprehension across
domains, the popular GPT-4o and command-R+ fail dramatically on protein
sequencing tasks. With the best performance at 32% there is much room for
improvement for all models. We hope that insights gained from CURIE can guide
the future development of LLMs in sciences. Evaluation code and data are in
https://github.com/google/curie
CURIE is a benchmark for evaluating Large Language Models (LLMs) in scientific problem-solving across various domains, highlighting areas of improvement for these models.
Large Language Models (LLMs) scientific problem-solving CURIE benchmark domain expertise comprehension multi-step reasoning protein sequencing tasks