SciCode: A Research Coding Benchmark Curated by Scientists
Since language models (LMs) now outperform average humans on many challenging
tasks, it has become increasingly difficult to develop challenging,
high-quality, and realistic evaluations. We address this issue by examining
LMs' capabilities to generate code for solving real scientific research
problems. Incorporating input from scientists and AI researchers in 16 diverse
natural science sub-fields, including mathematics, physics, chemistry, biology,
and materials science, we created a scientist-curated coding benchmark,
SciCode. The problems in SciCode naturally factorize into multiple subproblems,
each involving knowledge recall, reasoning, and code synthesis. In total,
SciCode contains 338 subproblems decomposed from 80 challenging main problems.
It offers optional descriptions specifying useful scientific background
information and scientist-annotated gold-standard solutions and test cases for
evaluation. Claude3.5-Sonnet, the best-performing model among those tested, can
solve only 4.6% of the problems in the most realistic setting. We believe that
SciCode demonstrates both contemporary LMs' progress towards becoming helpful
scientific assistants and sheds light on the development and evaluation of
scientific AI in the future.
669dcf3b7f28c43e09d5497d
Minyang Tian
extracted_pending
2024-07-22T03:17:16.720Z
64ead5f1213a0415bd22d0e4
/avatars/2797e96d93b999a3e5f816935eb43673.svg
Minyang Tian
user
mtian8
669dcf3b7f28c43e09d5497e
Luyu Gao
admin_assigned
2024-07-22T10:12:44.059Z
6009aa9126e23ab5edb47afd
/avatars/f8afa9cf648227ad5817323451ede378.svg
Luyu Gao
user
Luyu
669dcf3b7f28c43e09d5497f
Shizhuo Dylan Zhang
extracted_pending
2024-07-22T03:17:16.720Z
660ec5a2509153ca49775a7c
/avatars/97570fc245cc8ec7628da9c13bd35b71.svg
Hao Peng
user
haopeng01
669dcf3b7f28c43e09d54980
Xinan Chen
669dcf3b7f28c43e09d54981
Cunwei Fan
669dcf3b7f28c43e09d54982
Xuefei Guo
669dcf3b7f28c43e09d54983
Roland Haas
669dcf3b7f28c43e09d54984
Pan Ji
669dcf3b7f28c43e09d54985
Kittithat Krongchon
669dcf3b7f28c43e09d54986
Yao Li
669dcf3b7f28c43e09d54987
Shengyan Liu
admin_assigned
2024-07-22T10:31:26.403Z
661551375e8f1eaaba4c53b3
/avatars/1d41150d06ce4e4726269b9c48cbbc10.svg
LiuShengyang
user
LSY3579
669dcf3b7f28c43e09d54988
Di Luo
669dcf3b7f28c43e09d54989
Yutao Ma
admin_assigned
2024-07-22T10:30:46.216Z
630394e6eedc089484c367e5
/avatars/26560adc1466cad5b5646d1a2aea3c76.svg
MA YUTAO
user
mytkkb
669dcf3b7f28c43e09d5498a
Hao Tong
669dcf3b7f28c43e09d5498b
Kha Trinh
669dcf3b7f28c43e09d5498c
Chenyu Tian
admin_assigned
2024-07-22T10:13:53.923Z
624fa0a20f724e866aa833c6
/avatars/be97e25a08b9a69200d8c7cc9a756410.svg
Chenyu Tian
user
CYBruce
669dcf3b7f28c43e09d5498d
Zihan Wang
669dcf3b7f28c43e09d5498e
Bohao Wu
claimed_verified
2024-11-22T16:30:46.015Z
6717b1a510c78be6e4335dd4
/avatars/368e4af45151d2756726f45ec9cb0b64.svg
Bohao Wu
user
nlogn-27
669dcf3b7f28c43e09d5498f
Yanyu Xiong
admin_assigned
2024-07-22T10:28:56.221Z
645862428eef89b5c3c25ba1
/avatars/b46887a16d17c6e552d26a5a140e81e0.svg
Yanyu Xiong
user
yxiong5
669dcf3b7f28c43e09d54990
Shengzhu Yin
admin_assigned
2024-07-22T10:28:47.804Z
65306f8b2168c2bddd37930d
/avatars/3c476272977cc7d6241dc1d6d63bd377.svg
Shengzhu Yin
user
shengzhu
669dcf3b7f28c43e09d54991
Minhui Zhu
669dcf3b7f28c43e09d54992
Kilian Lieret
669dcf3b7f28c43e09d54993
Yanxin Lu
claimed_verified
2024-07-22T07:08:39.863Z
6610c90c6c59fff43622eef7
/avatars/f832a3918101aa5f0834f1456b154c37.svg
Yanxin Lu
user
amber1120
669dcf3b7f28c43e09d54994
Genglin Liu
admin_assigned
2024-07-22T10:16:25.059Z
64881deb8e004bb92b0f4845
/avatars/30a1e016d469bf7eb42c713351a9f65c.svg
Genglin Liu
user
genglinliu
669dcf3b7f28c43e09d54995
Yufeng Du
admin_assigned
2024-07-22T10:16:33.054Z
647e94f7770c299e56fc996f
/avatars/6b53e9ece40e1ebd4a5297b47ebd8b91.svg
Yufeng Du
user
yufeng16
669dcf3b7f28c43e09d54996
Tianhua Tao
admin_assigned
2024-07-22T10:16:40.367Z
61ecb698382b3b92fea7305f
/avatars/b1bcf0d9ade12172810cb65dc9c4e4d8.svg
Tianhua Tao
user
Tianhua
669dcf3b7f28c43e09d54997
Ofir Press
admin_assigned
2024-07-22T10:16:47.364Z
65b7a5657817e067a5ad45d8
/avatars/f8792352f16be80bd138ffc911138be1.svg
Ofir Press
user
ofirpress
669dcf3b7f28c43e09d54998
Jamie Callan
669dcf3b7f28c43e09d54999
Eliu Huerta
extracted_pending
2025-01-08T21:36:43.745Z
677eef9d26d2ab07cfbcf6c6
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/drlFgINAvi8M44TSME5-F.png
Eliu Huerta
user
elihu13
669dcf3b7f28c43e09d5499a
Hao Peng
admin_assigned
2024-07-22T10:18:14.706Z
660ec5a2509153ca49775a7c
/avatars/97570fc245cc8ec7628da9c13bd35b71.svg
Hao Peng
user
haopeng01
SciCode: A Research Coding Benchmark Curated by Scientists
Since language models (LMs) now outperform average humans on many challenging
tasks, it has become increasingly difficult to develop challenging,
high-quality, and realistic evaluations. We address this issue by examining
LMs' capabilities to generate code for solving real scientific research
problems. Incorporating input from scientists and AI researchers in 16 diverse
natural science sub-fields, including mathematics, physics, chemistry, biology,
and materials science, we created a scientist-curated coding benchmark,
SciCode. The problems in SciCode naturally factorize into multiple subproblems,
each involving knowledge recall, reasoning, and code synthesis. In total,
SciCode contains 338 subproblems decomposed from 80 challenging main problems.
It offers optional descriptions specifying useful scientific background
information and scientist-annotated gold-standard solutions and test cases for
evaluation. Claude3.5-Sonnet, the best-performing model among those tested, can
solve only 4.6% of the problems in the most realistic setting. We believe that
SciCode demonstrates both contemporary LMs' progress towards becoming helpful
scientific assistants and sheds light on the development and evaluation of
scientific AI in the future.
SciCode, a scientist-curated coding benchmark, evaluates language models' ability to solve scientific research problems across various fields, highlighting current capabilities and future challenges in AI-assisted science.
language models coding benchmark knowledge recall reasoning code synthesis scientific AI