RAGCap-Bench: Benchmarking Capabilities of LLMs in Agentic Retrieval
  Augmented Generation Systems
Retrieval-Augmented Generation (RAG) mitigates key limitations of Large
Language Models (LLMs)-such as factual errors, outdated knowledge, and
hallucinations-by dynamically retrieving external information. Recent work
extends this paradigm through agentic RAG systems, where LLMs act as agents to
iteratively plan, retrieve, and reason over complex queries. However, these
systems still struggle with challenging multi-hop questions, and their
intermediate reasoning capabilities remain underexplored. To address this, we
propose RAGCap-Bench, a capability-oriented benchmark for fine-grained
evaluation of intermediate tasks in agentic RAG workflows. We analyze outputs
from state-of-the-art systems to identify common tasks and the core
capabilities required for their execution, then construct a taxonomy of typical
LLM errors to design targeted evaluation questions. Experiments show that
"slow-thinking" models with stronger RAGCap performance achieve better
end-to-end results, underscoring the benchmark's validity and the importance of
enhancing these intermediate capabilities.
68f1b22b6e0bef323a68fdee
Jingru Lin
68f1b22b6e0bef323a68fdef
Chen Zhang
68f1b22b6e0bef323a68fdf0
Stephen Y. Liu
68f1b22b6e0bef323a68fdf1
Haizhou Li
RAGCap-Bench: Benchmarking Capabilities of LLMs in Agentic Retrieval
  Augmented Generation Systems
Retrieval-Augmented Generation (RAG) mitigates key limitations of Large
Language Models (LLMs)-such as factual errors, outdated knowledge, and
hallucinations-by dynamically retrieving external information. Recent work
extends this paradigm through agentic RAG systems, where LLMs act as agents to
iteratively plan, retrieve, and reason over complex queries. However, these
systems still struggle with challenging multi-hop questions, and their
intermediate reasoning capabilities remain underexplored. To address this, we
propose RAGCap-Bench, a capability-oriented benchmark for fine-grained
evaluation of intermediate tasks in agentic RAG workflows. We analyze outputs
from state-of-the-art systems to identify common tasks and the core
capabilities required for their execution, then construct a taxonomy of typical
LLM errors to design targeted evaluation questions. Experiments show that
"slow-thinking" models with stronger RAGCap performance achieve better
end-to-end results, underscoring the benchmark's validity and the importance of
enhancing these intermediate capabilities.
RAGCap-Bench evaluates intermediate tasks in agentic RAG workflows, highlighting the importance of enhancing these capabilities for better end-to-end performance.
Retrieval-Augmented Generation RAG Large Language Models LLMs agentic RAG multi-hop questions intermediate reasoning capability-oriented benchmark RAGCap-Bench slow-thinking models RAGCap performance