Understanding DeepResearch via Reports
DeepResearch agents represent a transformative AI paradigm, conducting
expert-level research through sophisticated reasoning and multi-tool
integration. However, evaluating these systems remains critically challenging
due to open-ended research scenarios and existing benchmarks that focus on
isolated capabilities rather than holistic performance. Unlike traditional LLM
tasks, DeepResearch systems must synthesize diverse sources, generate insights,
and present coherent findings, which are capabilities that resist simple
verification. To address this gap, we introduce DeepResearch-ReportEval, a
comprehensive framework designed to assess DeepResearch systems through their
most representative outputs: research reports. Our approach systematically
measures three dimensions: quality, redundancy, and factuality, using an
innovative LLM-as-a-Judge methodology achieving strong expert concordance. We
contribute a standardized benchmark of 100 curated queries spanning 12
real-world categories, enabling systematic capability comparison. Our
evaluation of four leading commercial systems reveals distinct design
philosophies and performance trade-offs, establishing foundational insights as
DeepResearch evolves from information assistants toward intelligent research
partners. Source code and data are available at:
https://github.com/HKUDS/DeepResearch-Eval.
68ec6ff6cd07fb414898c9c8
Tianyu Fan
claimed_verified
2025-10-13T10:05:27.374Z
65f3ee4efe1935d2e9d4a678
/avatars/ffd9f945183b4c890f885c82739c28a6.svg
Tianyu Fan
user
T1anyu
68ec6ff6cd07fb414898c9c9
Xinyao Niu
68ec6ff6cd07fb414898c9ca
Yuxiang Zheng
68ec6ff6cd07fb414898c9cb
Fengji Zhang
68ec6ff6cd07fb414898c9cc
Chengen Huang
68ec6ff6cd07fb414898c9cd
Bei Chen
68ec6ff6cd07fb414898c9ce
Junyang Lin
68ec6ff6cd07fb414898c9cf
Chao Huang
Understanding DeepResearch via Reports
DeepResearch agents represent a transformative AI paradigm, conducting
expert-level research through sophisticated reasoning and multi-tool
integration. However, evaluating these systems remains critically challenging
due to open-ended research scenarios and existing benchmarks that focus on
isolated capabilities rather than holistic performance. Unlike traditional LLM
tasks, DeepResearch systems must synthesize diverse sources, generate insights,
and present coherent findings, which are capabilities that resist simple
verification. To address this gap, we introduce DeepResearch-ReportEval, a
comprehensive framework designed to assess DeepResearch systems through their
most representative outputs: research reports. Our approach systematically
measures three dimensions: quality, redundancy, and factuality, using an
innovative LLM-as-a-Judge methodology achieving strong expert concordance. We
contribute a standardized benchmark of 100 curated queries spanning 12
real-world categories, enabling systematic capability comparison. Our
evaluation of four leading commercial systems reveals distinct design
philosophies and performance trade-offs, establishing foundational insights as
DeepResearch evolves from information assistants toward intelligent research
partners. Source code and data are available at:
https://github.com/HKUDS/DeepResearch-Eval.
A framework evaluates DeepResearch systems by assessing the quality, redundancy, and factuality of their research reports using an LLM-as-a-Judge methodology.
DeepResearch LLM research reports LLM-as-a-Judge expert concordance standardized benchmark real-world categories intelligent research partners