Demystifying deep search: a holistic evaluation with hint-free multi-hop
  questions and factorised metrics
RAG (Retrieval-Augmented Generation) systems and web agents are increasingly
evaluated on multi-hop deep search tasks, yet current practice suffers from two
major limitations. First, most benchmarks leak the reasoning path in the
question text, allowing models to follow surface cues rather than discover
reasoning chains autonomously. Second, evaluation is typically reduced to a
single pass rate, which collapses diverse behaviours into one score and
obscures whether failures stem from inadequate search, poor knowledge use, or
inappropriate refusal. To address these issues, we present WebDetective, a
benchmark of hint-free multi-hop questions paired with a controlled Wikipedia
sandbox that ensures full traceability of model actions, and a holistic
evaluation framework that separates search sufficiency, knowledge utilisation,
and refusal behaviour. Our evaluation of 25 state-of-the-art models reveals
systematic weaknesses across all architectures: models struggle with knowledge
utilisation despite having sufficient evidence and demonstrate near-absent
appropriate refusal when evidence is lacking. These patterns expose a
fundamental gap: today's systems excel at executing given reasoning paths but
fail when required to discover them. We develop an agentic workflow,
EvidenceLoop, that explicitly targets the challenges our benchmark identifies,
incorporating verification loops and systematic evidence tracking that improve
both search and synthesis capabilities. This baseline demonstrates that
WebDetective's diagnostic framework can guide concrete architectural
improvements, establishing our benchmark as a critical tool for developing
genuinely autonomous reasoning systems rather than pattern-following agents.
68e5bde7975ac4c405ef2031
Maojia Song
68e5bde7975ac4c405ef2032
Renhang Liu
68e5bde7975ac4c405ef2033
Xinyu Wang
68e5bde7975ac4c405ef2034
Yong Jiang
68e5bde7975ac4c405ef2035
Pengjun Xie
68e5bde7975ac4c405ef2036
Fei Huang
68e5bde7975ac4c405ef2037
Soujanya Poria
68e5bde7975ac4c405ef2038
Jingren Zhou
Demystifying deep search: a holistic evaluation with hint-free multi-hop
  questions and factorised metrics
RAG (Retrieval-Augmented Generation) systems and web agents are increasingly
evaluated on multi-hop deep search tasks, yet current practice suffers from two
major limitations. First, most benchmarks leak the reasoning path in the
question text, allowing models to follow surface cues rather than discover
reasoning chains autonomously. Second, evaluation is typically reduced to a
single pass rate, which collapses diverse behaviours into one score and
obscures whether failures stem from inadequate search, poor knowledge use, or
inappropriate refusal. To address these issues, we present WebDetective, a
benchmark of hint-free multi-hop questions paired with a controlled Wikipedia
sandbox that ensures full traceability of model actions, and a holistic
evaluation framework that separates search sufficiency, knowledge utilisation,
and refusal behaviour. Our evaluation of 25 state-of-the-art models reveals
systematic weaknesses across all architectures: models struggle with knowledge
utilisation despite having sufficient evidence and demonstrate near-absent
appropriate refusal when evidence is lacking. These patterns expose a
fundamental gap: today's systems excel at executing given reasoning paths but
fail when required to discover them. We develop an agentic workflow,
EvidenceLoop, that explicitly targets the challenges our benchmark identifies,
incorporating verification loops and systematic evidence tracking that improve
both search and synthesis capabilities. This baseline demonstrates that
WebDetective's diagnostic framework can guide concrete architectural
improvements, establishing our benchmark as a critical tool for developing
genuinely autonomous reasoning systems rather than pattern-following agents.
WebDetective is a benchmark for evaluating multi-hop reasoning in RAG systems and web agents, addressing issues of reasoning path leakage and single-pass evaluation, and introducing a framework to improve knowledge utilization and refusal behavior.
RAG Retrieval-Augmented Generation multi-hop deep search reasoning path knowledge utilisation refusal behaviour Wikipedia sandbox EvidenceLoop verification loops evidence tracking