BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of
  Deep-Research Agent
Deep-Research agents, which integrate large language models (LLMs) with
search tools, have shown success in improving the effectiveness of handling
complex queries that require iterative search planning and reasoning over
search results. Evaluations on current benchmarks like BrowseComp relies on
black-box live web search APIs, have notable limitations in (1) fairness:
dynamic and opaque web APIs hinder fair comparisons and reproducibility of deep
research methods; (2) transparency: lack of control over the document corpus
makes it difficult to isolate retriever contributions. In other words, the
current evaluations may compare a complete deep research system at a given
time, but they do not foster well-controlled experiments to provide insights
into the capability of underlying deep research LLMs. To address these
challenges, we introduce BrowseComp-Plus, a benchmark derived from BrowseComp,
employing a fixed, carefully curated corpus. Each query in BrowseComp-Plus
includes human-verified supporting documents and mined challenging negatives,
enabling controlled experimentation. The benchmark is shown to be effective in
distinguishing the performance of deep research systems. For instance, the
open-source model Search-R1, when paired with the BM25 retriever, achieves
3.86% accuracy, whereas the GPT-5 achieves 55.9%. Integrating the GPT-5 with
the Qwen3-Embedding-8B retriever further enhances its accuracy to 70.1% with
fewer search calls. This benchmark allows comprehensive evaluation and
disentangled analysis of deep research agents and retrieval methods, fostering
insights into retrieval effectiveness, citation accuracy, and context
engineering in Deep-Research system.
689aa99dfab6fdd2e52ac443
Zijian Chen
claimed_verified
2025-08-22T07:24:52.373Z
644db73976c0ab1880b8bc74
https://cdn-avatars.huggingface.co/v1/production/uploads/644db73976c0ab1880b8bc74/ZRjLlwMHKyfNFDewTzjxj.png
Steven Chen
user
s42chen
689aa99dfab6fdd2e52ac444
Xueguang Ma
claimed_verified
2025-08-13T07:21:03.660Z
5ec82854968f6028e0559f70
/avatars/45b58d912f7d00cb351947cd79d5eeb4.svg
Xueguang Ma
user
MrLight
689aa99dfab6fdd2e52ac445
Shengyao Zhuang
claimed_verified
2025-08-13T07:21:06.722Z
60d97add9fe99457e2010efe
https://cdn-avatars.huggingface.co/v1/production/uploads/1624866593786-60d97add9fe99457e2010efe.png
Shengyao Zhuang
user
ArvinZhuang
689aa99dfab6fdd2e52ac446
Ping Nie
claimed_verified
2025-08-13T07:21:00.298Z
65358802a920f38780b3248a
/avatars/9415510b598079973c2b0436ad12db9c.svg
Ping Nie
user
pingnieuk
689aa99dfab6fdd2e52ac447
Kai Zou
689aa99dfab6fdd2e52ac448
Andrew Liu
689aa99dfab6fdd2e52ac449
Joshua Green
689aa99dfab6fdd2e52ac44a
Kshama Patel
689aa99dfab6fdd2e52ac44b
Ruoxi Meng
689aa99dfab6fdd2e52ac44c
Mingyi Su
689aa99dfab6fdd2e52ac44d
Sahel Sharifymoghaddam
689aa99dfab6fdd2e52ac44e
Yanxi Li
689aa99dfab6fdd2e52ac44f
Haoran Hong
689aa99dfab6fdd2e52ac450
Xinyu Shi
689aa99dfab6fdd2e52ac451
Xuye Liu
claimed_verified
2025-08-18T06:59:20.432Z
60845b6ca5da133ac6c38681
/avatars/01dfcf615a57c37ff19276d79f423cf1.svg
Xuye Liu
user
richard
689aa99dfab6fdd2e52ac452
Nandan Thakur
claimed_verified
2025-08-13T07:20:57.483Z
60196690dd31fde3c1062960
https://cdn-avatars.huggingface.co/v1/production/uploads/1612277330660-noauth.jpeg
Nandan Thakur
user
nthakur
689aa99dfab6fdd2e52ac453
Crystina Zhang
689aa99dfab6fdd2e52ac454
Luyu Gao
689aa99dfab6fdd2e52ac455
Wenhu Chen
689aa99dfab6fdd2e52ac456
Jimmy Lin
BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of
  Deep-Research Agent
Deep-Research agents, which integrate large language models (LLMs) with
search tools, have shown success in improving the effectiveness of handling
complex queries that require iterative search planning and reasoning over
search results. Evaluations on current benchmarks like BrowseComp relies on
black-box live web search APIs, have notable limitations in (1) fairness:
dynamic and opaque web APIs hinder fair comparisons and reproducibility of deep
research methods; (2) transparency: lack of control over the document corpus
makes it difficult to isolate retriever contributions. In other words, the
current evaluations may compare a complete deep research system at a given
time, but they do not foster well-controlled experiments to provide insights
into the capability of underlying deep research LLMs. To address these
challenges, we introduce BrowseComp-Plus, a benchmark derived from BrowseComp,
employing a fixed, carefully curated corpus. Each query in BrowseComp-Plus
includes human-verified supporting documents and mined challenging negatives,
enabling controlled experimentation. The benchmark is shown to be effective in
distinguishing the performance of deep research systems. For instance, the
open-source model Search-R1, when paired with the BM25 retriever, achieves
3.86% accuracy, whereas the GPT-5 achieves 55.9%. Integrating the GPT-5 with
the Qwen3-Embedding-8B retriever further enhances its accuracy to 70.1% with
fewer search calls. This benchmark allows comprehensive evaluation and
disentangled analysis of deep research agents and retrieval methods, fostering
insights into retrieval effectiveness, citation accuracy, and context
engineering in Deep-Research system.
BrowseComp-Plus, a curated benchmark, enables controlled evaluation of deep research agents and retrieval methods, providing insights into their performance and effectiveness.
deep-Research agents large language models (LLMs) search tools iterative search planning BrowseComp black-box live web search APIs fairness transparency document corpus retriever contributions BrowseComp-Plus human-verified supporting documents challenging negatives Search-R1 GPT-5 Qwen3-Embedding-8B retrieval effectiveness citation accuracy context engineering