Hop, Skip, and Overthink: Diagnosing Why Reasoning Models Fumble during
  Multi-Hop Analysis
The emergence of reasoning models and their integration into practical AI
chat bots has led to breakthroughs in solving advanced math, deep search, and
extractive question answering problems that requires a complex and multi-step
thought process. Yet, a complete understanding of why these models hallucinate
more than general purpose language models is missing. In this investigative
study, we systematicallyexplore reasoning failures of contemporary language
models on multi-hop question answering tasks. We introduce a novel, nuanced
error categorization framework that examines failures across three critical
dimensions: the diversity and uniqueness of source documents involved ("hops"),
completeness in capturing relevant information ("coverage"), and cognitive
inefficiency ("overthinking"). Through rigorous hu-man annotation, supported by
complementary automated metrics, our exploration uncovers intricate error
patterns often hidden by accuracy-centric evaluations. This investigative
approach provides deeper insights into the cognitive limitations of current
models and offers actionable guidance toward enhancing reasoning fidelity,
transparency, and robustness in future language modeling efforts.
68955ed748b0ae5ca2710d36
Anushka Yadav
68955ed748b0ae5ca2710d37
Isha Nalawade
68955ed748b0ae5ca2710d38
Srujana Pillarichety
68955ed748b0ae5ca2710d39
Yashwanth Babu
68955ed748b0ae5ca2710d3a
Reshmi Ghosh
claimed_verified
2025-08-08T16:19:46.570Z
63228cad95d6f717a8c8e1ec
/avatars/5fa606f63f841879af1a7617366d18b4.svg
Reshmi Ghosh
user
reshmighosh
68955ed748b0ae5ca2710d3b
Samyadeep Basu
68955ed748b0ae5ca2710d3c
Wenlong Zhao
68955ed748b0ae5ca2710d3d
Ali Nasaeh
68955ed748b0ae5ca2710d3e
Sriram Balasubramanian
68955ed748b0ae5ca2710d3f
Soundararajan Srinivasan
Hop, Skip, and Overthink: Diagnosing Why Reasoning Models Fumble during
  Multi-Hop Analysis
The emergence of reasoning models and their integration into practical AI
chat bots has led to breakthroughs in solving advanced math, deep search, and
extractive question answering problems that requires a complex and multi-step
thought process. Yet, a complete understanding of why these models hallucinate
more than general purpose language models is missing. In this investigative
study, we systematicallyexplore reasoning failures of contemporary language
models on multi-hop question answering tasks. We introduce a novel, nuanced
error categorization framework that examines failures across three critical
dimensions: the diversity and uniqueness of source documents involved ("hops"),
completeness in capturing relevant information ("coverage"), and cognitive
inefficiency ("overthinking"). Through rigorous hu-man annotation, supported by
complementary automated metrics, our exploration uncovers intricate error
patterns often hidden by accuracy-centric evaluations. This investigative
approach provides deeper insights into the cognitive limitations of current
models and offers actionable guidance toward enhancing reasoning fidelity,
transparency, and robustness in future language modeling efforts.
Research investigates reasoning failures in language models for multi-hop question answering, introducing a framework to categorize errors and improve model fidelity.
reasoning models multi-hop question answering error categorization cognitive limitations reasoning fidelity transparency robustness