ResearchBench: Benchmarking LLMs in Scientific Discovery via
  Inspiration-Based Task Decomposition
Large language models (LLMs) have demonstrated potential in assisting
scientific research, yet their ability to discover high-quality research
hypotheses remains unexamined due to the lack of a dedicated benchmark. To
address this gap, we introduce the first large-scale benchmark for evaluating
LLMs with a near-sufficient set of sub-tasks of scientific discovery:
inspiration retrieval, hypothesis composition, and hypothesis ranking. We
develop an automated framework that extracts critical components - research
questions, background surveys, inspirations, and hypotheses - from scientific
papers across 12 disciplines, with expert validation confirming its accuracy.
To prevent data contamination, we focus exclusively on papers published in
2024, ensuring minimal overlap with LLM pretraining data. Our evaluation
reveals that LLMs perform well in retrieving inspirations, an
out-of-distribution task, suggesting their ability to surface novel knowledge
associations. This positions LLMs as "research hypothesis mines", capable of
facilitating automated scientific discovery by generating innovative hypotheses
at scale with minimal human intervention.
67e63581e8a16f38741f4baa
Yujie Liu
admin_assigned
2025-03-28T08:58:09.374Z
661980b46a43b2760d8d551f
/avatars/e4743d55303fe3a88688c29dd4c67a69.svg
Yujie Liu
user
yujieliu
67e63581e8a16f38741f4bab
Zonglin Yang
admin_assigned
2025-03-28T08:58:40.651Z
646a11791556443f24b582e9
/avatars/b54d416ddca2f124492ba51bc4b95fd2.svg
Zonglin Yang
user
ZonglinY
67e63581e8a16f38741f4bac
Tong Xie
67e63581e8a16f38741f4bad
Jinjie Ni
admin_assigned
2025-03-28T08:58:47.888Z
62a7362fd1e7a011fd4e31a7
https://cdn-avatars.huggingface.co/v1/production/uploads/62a7362fd1e7a011fd4e31a7/ZY_mwH-sI0o05SHpLqwc7.png
Jinjie Ni
user
jinjieni
67e63581e8a16f38741f4bae
Ben Gao
admin_assigned
2025-03-28T08:58:54.480Z
64fe3c6b4c8924c4fed6d97b
/avatars/5e0a49372af19aae9ec5ee84b299d111.svg
Ben Gao
user
bgao22182
67e63581e8a16f38741f4baf
Yuqiang Li
admin_assigned
2025-03-28T08:59:07.268Z
65cb61ba0390fce629de99d1
/avatars/9b4b4556ffd8de215dc37b02366d781b.svg
Yuqiang Li
user
yuqiangli
67e63581e8a16f38741f4bb0
Shixiang Tang
admin_assigned
2025-03-28T08:59:12.700Z
6436403bf3b08e267d9f0329
/avatars/a5d9c3d47073e71e4cea124d9c17356d.svg
SHIXIANG TANG
user
tangshixiang
67e63581e8a16f38741f4bb1
Wanli Ouyang
67e63581e8a16f38741f4bb2
Erik Cambria
67e63581e8a16f38741f4bb3
Dongzhan Zhou
admin_assigned
2025-03-28T08:59:22.218Z
6538b861613fe158bd581e35
/avatars/6817dbfe903675721fd227058b0a91ac.svg
Dongzhan Zhou
user
schrodingers-tiger
ResearchBench: Benchmarking LLMs in Scientific Discovery via
  Inspiration-Based Task Decomposition
Large language models (LLMs) have demonstrated potential in assisting
scientific research, yet their ability to discover high-quality research
hypotheses remains unexamined due to the lack of a dedicated benchmark. To
address this gap, we introduce the first large-scale benchmark for evaluating
LLMs with a near-sufficient set of sub-tasks of scientific discovery:
inspiration retrieval, hypothesis composition, and hypothesis ranking. We
develop an automated framework that extracts critical components - research
questions, background surveys, inspirations, and hypotheses - from scientific
papers across 12 disciplines, with expert validation confirming its accuracy.
To prevent data contamination, we focus exclusively on papers published in
2024, ensuring minimal overlap with LLM pretraining data. Our evaluation
reveals that LLMs perform well in retrieving inspirations, an
out-of-distribution task, suggesting their ability to surface novel knowledge
associations. This positions LLMs as "research hypothesis mines", capable of
facilitating automated scientific discovery by generating innovative hypotheses
at scale with minimal human intervention.
LLMs demonstrate proficiency in inspiration retrieval using a newly developed benchmark, showing potential in automated scientific hypothesis generation.
large language models LLMs scientific discovery benchmark sub-tasks inspiration retrieval hypothesis composition hypothesis ranking research questions background surveys inspirations hypotheses automated framework out-of-distribution task automated scientific discovery innovative hypotheses