ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks
The advent of Deep Research agents has substantially reduced the time
required for conducting extensive research tasks. However, these tasks
inherently demand rigorous standards of factual accuracy and comprehensiveness,
necessitating thorough evaluation before widespread adoption. In this paper, we
propose ReportBench, a systematic benchmark designed to evaluate the content
quality of research reports generated by large language models (LLMs). Our
evaluation focuses on two critical dimensions: (1) the quality and relevance of
cited literature, and (2) the faithfulness and veracity of the statements
within the generated reports. ReportBench leverages high-quality published
survey papers available on arXiv as gold-standard references, from which we
apply reverse prompt engineering to derive domain-specific prompts and
establish a comprehensive evaluation corpus. Furthermore, we develop an
agent-based automated framework within ReportBench that systematically analyzes
generated reports by extracting citations and statements, checking the
faithfulness of cited content against original sources, and validating
non-cited claims using web-based resources. Empirical evaluations demonstrate
that commercial Deep Research agents such as those developed by OpenAI and
Google consistently generate more comprehensive and reliable reports than
standalone LLMs augmented with search or browsing tools. However, there remains
substantial room for improvement in terms of the breadth and depth of research
coverage, as well as factual consistency. The complete code and data will be
released at the following link: https://github.com/ByteDance-BandAI/ReportBench
68ae8365364411bea07df821
Minghao Li
claimed_verified
2025-09-18T13:28:09.486Z
5f574f9e8bf55658acfed37e
https://cdn-avatars.huggingface.co/v1/production/uploads/1637285694180-5f574f9e8bf55658acfed37e.jpeg
Minghao Li
user
liminghao1630
68ae8365364411bea07df822
Ying Zeng
68ae8365364411bea07df823
Zhihao Cheng
68ae8365364411bea07df824
Cong Ma
68ae8365364411bea07df825
Kai Jia
ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks
The advent of Deep Research agents has substantially reduced the time
required for conducting extensive research tasks. However, these tasks
inherently demand rigorous standards of factual accuracy and comprehensiveness,
necessitating thorough evaluation before widespread adoption. In this paper, we
propose ReportBench, a systematic benchmark designed to evaluate the content
quality of research reports generated by large language models (LLMs). Our
evaluation focuses on two critical dimensions: (1) the quality and relevance of
cited literature, and (2) the faithfulness and veracity of the statements
within the generated reports. ReportBench leverages high-quality published
survey papers available on arXiv as gold-standard references, from which we
apply reverse prompt engineering to derive domain-specific prompts and
establish a comprehensive evaluation corpus. Furthermore, we develop an
agent-based automated framework within ReportBench that systematically analyzes
generated reports by extracting citations and statements, checking the
faithfulness of cited content against original sources, and validating
non-cited claims using web-based resources. Empirical evaluations demonstrate
that commercial Deep Research agents such as those developed by OpenAI and
Google consistently generate more comprehensive and reliable reports than
standalone LLMs augmented with search or browsing tools. However, there remains
substantial room for improvement in terms of the breadth and depth of research
coverage, as well as factual consistency. The complete code and data will be
released at the following link: https://github.com/ByteDance-BandAI/ReportBench
ReportBench evaluates the content quality of research reports generated by large language models, focusing on cited literature quality and statement faithfulness, demonstrating that commercial Deep Research agents produce more comprehensive and reliable reports than standalone LLMs.
Deep Research agents large language models ReportBench reverse prompt engineering evaluation corpus agent-based automated framework citations statements faithfulness factual consistency