DeepScholar-Bench: A Live Benchmark and Automated Evaluation for
  Generative Research Synthesis
The ability to research and synthesize knowledge is central to human
expertise and progress. An emerging class of systems promises these exciting
capabilities through generative research synthesis, performing retrieval over
the live web and synthesizing discovered sources into long-form, cited
summaries. However, evaluating such systems remains an open challenge: existing
question-answering benchmarks focus on short-form factual responses, while
expert-curated datasets risk staleness and data contamination. Both fail to
capture the complexity and evolving nature of real research synthesis tasks. In
this work, we introduce DeepScholar-bench, a live benchmark and holistic,
automated evaluation framework designed to evaluate generative research
synthesis. DeepScholar-bench draws queries from recent, high-quality ArXiv
papers and focuses on a real research synthesis task: generating the related
work sections of a paper by retrieving, synthesizing, and citing prior
research. Our evaluation framework holistically assesses performance across
three key dimensions, knowledge synthesis, retrieval quality, and
verifiability. We also develop DeepScholar-base, a reference pipeline
implemented efficiently using the LOTUS API. Using the DeepScholar-bench
framework, we perform a systematic evaluation of prior open-source systems,
search AI's, OpenAI's DeepResearch, and DeepScholar-base. We find that
DeepScholar-base establishes a strong baseline, attaining competitive or higher
performance than each other method. We also find that DeepScholar-bench remains
far from saturated, with no system exceeding a score of 19% across all
metrics. These results underscore the difficulty of DeepScholar-bench, as well
as its importance for progress towards AI systems capable of generative
research synthesis. We make our code available at
https://github.com/guestrin-lab/deepscholar-bench.
68afd4f8245176306494cc9d
Liana Patel
claimed_verified
2025-09-12T16:21:55.376Z
659af8d4e6df49a09d2b1d5e
/avatars/dcd3a8e8a3323bf723b17098ee215419.svg
LIana
user
liana313
68afd4f8245176306494cc9e
Negar Arabzadeh
admin_assigned
2025-08-28T18:57:39.579Z
5f3741e37e58354338621752
/avatars/b2337760c33ccca1d594824ae7353a8a.svg
Negar Arabzadeh
user
narabzad
68afd4f8245176306494cc9f
Harshit Gupta
68afd4f8245176306494cca0
Ankita Sundar
68afd4f8245176306494cca1
Ion Stoica
68afd4f8245176306494cca2
Matei Zaharia
admin_assigned
2025-08-28T18:57:59.132Z
642509c6d476e4ad5568dd31
https://cdn-avatars.huggingface.co/v1/production/uploads/642509c6d476e4ad5568dd31/dPiEV70hMeuY-cveLNPQg.jpeg
Matei Zaharia
user
mateiz
68afd4f8245176306494cca3
Carlos Guestrin
admin_assigned
2025-08-28T18:58:04.940Z
677c45511fbb93b90f1c6f3d
/avatars/3a78fdd8d1debc8d267e80e4b7a6bf77.svg
Carlos Guestrin
user
guestrin
DeepScholar-Bench: A Live Benchmark and Automated Evaluation for
  Generative Research Synthesis
The ability to research and synthesize knowledge is central to human
expertise and progress. An emerging class of systems promises these exciting
capabilities through generative research synthesis, performing retrieval over
the live web and synthesizing discovered sources into long-form, cited
summaries. However, evaluating such systems remains an open challenge: existing
question-answering benchmarks focus on short-form factual responses, while
expert-curated datasets risk staleness and data contamination. Both fail to
capture the complexity and evolving nature of real research synthesis tasks. In
this work, we introduce DeepScholar-bench, a live benchmark and holistic,
automated evaluation framework designed to evaluate generative research
synthesis. DeepScholar-bench draws queries from recent, high-quality ArXiv
papers and focuses on a real research synthesis task: generating the related
work sections of a paper by retrieving, synthesizing, and citing prior
research. Our evaluation framework holistically assesses performance across
three key dimensions, knowledge synthesis, retrieval quality, and
verifiability. We also develop DeepScholar-base, a reference pipeline
implemented efficiently using the LOTUS API. Using the DeepScholar-bench
framework, we perform a systematic evaluation of prior open-source systems,
search AI's, OpenAI's DeepResearch, and DeepScholar-base. We find that
DeepScholar-base establishes a strong baseline, attaining competitive or higher
performance than each other method. We also find that DeepScholar-bench remains
far from saturated, with no system exceeding a score of 19% across all
metrics. These results underscore the difficulty of DeepScholar-bench, as well
as its importance for progress towards AI systems capable of generative
research synthesis. We make our code available at
https://github.com/guestrin-lab/deepscholar-bench.
DeepScholar-bench evaluates generative research synthesis systems by assessing their performance in creating related work sections from recent ArXiv papers, focusing on knowledge synthesis, retrieval quality, and verifiability.
generative research synthesis DeepScholar-bench retrieval quality verifiability knowledge synthesis DeepScholar-base LOTUS API search AI OpenAI's DeepResearch