DeepResearchGym: A Free, Transparent, and Reproducible Evaluation
  Sandbox for Deep Research
Deep research systems represent an emerging class of agentic information
retrieval methods that generate comprehensive and well-supported reports to
complex queries. However, most existing frameworks rely on dynamic commercial
search APIs, which pose reproducibility and transparency challenges in addition
to their cost. To address these limitations, we introduce DeepResearchGym, an
open-source sandbox that combines a reproducible search API with a rigorous
evaluation protocol for benchmarking deep research systems. The API indexes
large-scale public web corpora, namely ClueWeb22 and FineWeb, using a
state-of-the-art dense retriever and approximate nearest neighbor search via
DiskANN. It achieves lower latency than popular commercial APIs while ensuring
stable document rankings across runs, and is freely available for research use.
To evaluate deep research systems' outputs, we extend the Researchy Questions
benchmark with automatic metrics through LLM-as-a-judge assessments to measure
alignment with users' information needs, retrieval faithfulness, and report
quality. Experimental results show that systems integrated with DeepResearchGym
achieve performance comparable to those using commercial APIs, with performance
rankings remaining consistent across evaluation metrics. A human evaluation
study further confirms that our automatic protocol aligns with human
preferences, validating the framework's ability to help support controlled
assessment of deep research systems. Our code and API documentation are
available at https://www.deepresearchgym.ai.
6837bc8d0b39c9653de4d06f
João Coelho
claimed_verified
2025-05-29T07:42:45.049Z
6244451c9fdefb55a0b900cc
/avatars/ca2b46ddb5d905501d827920582b5438.svg
Joao Coelho
user
jmvcoelho
6837bc8d0b39c9653de4d070
Jingjie Ning
claimed_verified
2025-05-29T20:18:58.772Z
6310a812a23f0327bce68778
https://cdn-avatars.huggingface.co/v1/production/uploads/6310a812a23f0327bce68778/dp9JherZV-RYiSRKnvxsh.jpeg
Ethan Ning
user
ethanning
6837bc8d0b39c9653de4d071
Jingyuan He
6837bc8d0b39c9653de4d072
Kangrui Mao
6837bc8d0b39c9653de4d073
Abhijay Paladugu
6837bc8d0b39c9653de4d074
Pranav Setlur
6837bc8d0b39c9653de4d075
Jiahe Jin
6837bc8d0b39c9653de4d076
Jamie Callan
6837bc8d0b39c9653de4d077
João Magalhães
6837bc8d0b39c9653de4d078
Bruno Martins
6837bc8d0b39c9653de4d079
Chenyan Xiong
DeepResearchGym: A Free, Transparent, and Reproducible Evaluation
  Sandbox for Deep Research
Deep research systems represent an emerging class of agentic information
retrieval methods that generate comprehensive and well-supported reports to
complex queries. However, most existing frameworks rely on dynamic commercial
search APIs, which pose reproducibility and transparency challenges in addition
to their cost. To address these limitations, we introduce DeepResearchGym, an
open-source sandbox that combines a reproducible search API with a rigorous
evaluation protocol for benchmarking deep research systems. The API indexes
large-scale public web corpora, namely ClueWeb22 and FineWeb, using a
state-of-the-art dense retriever and approximate nearest neighbor search via
DiskANN. It achieves lower latency than popular commercial APIs while ensuring
stable document rankings across runs, and is freely available for research use.
To evaluate deep research systems' outputs, we extend the Researchy Questions
benchmark with automatic metrics through LLM-as-a-judge assessments to measure
alignment with users' information needs, retrieval faithfulness, and report
quality. Experimental results show that systems integrated with DeepResearchGym
achieve performance comparable to those using commercial APIs, with performance
rankings remaining consistent across evaluation metrics. A human evaluation
study further confirms that our automatic protocol aligns with human
preferences, validating the framework's ability to help support controlled
assessment of deep research systems. Our code and API documentation are
available at https://www.deepresearchgym.ai.
DeepResearchGym provides an open-source evaluation framework for deep research systems using a reproducible search API and LLM-as-a-judge assessments.
agentic information retrieval deep research systems search API reproducibility transparency open-source sandbox ClueWeb22 FineWeb dense retriever approximate nearest neighbor search DiskANN Researchy Questions benchmark LLM-as-a-judge retrieval faithfulness report quality human evaluation