Are We on the Right Way for Assessing Document Retrieval-Augmented
  Generation?
Retrieval-Augmented Generation (RAG) systems using Multimodal Large Language
Models (MLLMs) show great promise for complex document understanding, yet their
development is critically hampered by inadequate evaluation. Current benchmarks
often focus on specific part of document RAG system and use synthetic data with
incomplete ground truth and evidence labels, therefore failing to reflect
real-world bottlenecks and challenges. To overcome these limitations, we
introduce Double-Bench: a new large-scale, multilingual, and multimodal
evaluation system that is able to produce fine-grained assessment to each
component within document RAG systems. It comprises 3,276 documents (72,880
pages) and 5,168 single- and multi-hop queries across 6 languages and 4
document types with streamlined dynamic update support for potential data
contamination issues. Queries are grounded in exhaustively scanned evidence
pages and verified by human experts to ensure maximum quality and completeness.
Our comprehensive experiments across 9 state-of-the-art embedding models, 4
MLLMs and 4 end-to-end document RAG frameworks demonstrate the gap between text
and visual embedding models is narrowing, highlighting the need in building
stronger document retrieval models. Our findings also reveal the
over-confidence dilemma within current document RAG frameworks that tend to
provide answer even without evidence support. We hope our fully open-source
Double-Bench provide a rigorous foundation for future research in advanced
document RAG systems. We plan to retrieve timely corpus and release new
benchmarks on an annual basis.
6892c4f18da45ffb0a2b2440
Wenxuan Shen
6892c4f18da45ffb0a2b2441
Mingjia Wang
6892c4f18da45ffb0a2b2442
Yaochen Wang
6892c4f18da45ffb0a2b2443
Dongping Chen
6892c4f18da45ffb0a2b2444
Junjie Yang
6892c4f18da45ffb0a2b2445
Yao Wan
6892c4f18da45ffb0a2b2446
Weiwei Lin
Are We on the Right Way for Assessing Document Retrieval-Augmented
  Generation?
Retrieval-Augmented Generation (RAG) systems using Multimodal Large Language
Models (MLLMs) show great promise for complex document understanding, yet their
development is critically hampered by inadequate evaluation. Current benchmarks
often focus on specific part of document RAG system and use synthetic data with
incomplete ground truth and evidence labels, therefore failing to reflect
real-world bottlenecks and challenges. To overcome these limitations, we
introduce Double-Bench: a new large-scale, multilingual, and multimodal
evaluation system that is able to produce fine-grained assessment to each
component within document RAG systems. It comprises 3,276 documents (72,880
pages) and 5,168 single- and multi-hop queries across 6 languages and 4
document types with streamlined dynamic update support for potential data
contamination issues. Queries are grounded in exhaustively scanned evidence
pages and verified by human experts to ensure maximum quality and completeness.
Our comprehensive experiments across 9 state-of-the-art embedding models, 4
MLLMs and 4 end-to-end document RAG frameworks demonstrate the gap between text
and visual embedding models is narrowing, highlighting the need in building
stronger document retrieval models. Our findings also reveal the
over-confidence dilemma within current document RAG frameworks that tend to
provide answer even without evidence support. We hope our fully open-source
Double-Bench provide a rigorous foundation for future research in advanced
document RAG systems. We plan to retrieve timely corpus and release new
benchmarks on an annual basis.
Double-Bench is a large-scale, multilingual, and multimodal evaluation system for document Retrieval-Augmented Generation (RAG) systems, addressing limitations in current benchmarks and providing comprehensive assessments of system components.
Retrieval-Augmented Generation Multimodal Large Language Models document RAG system Double-Bench large-scale multilingual multimodal evaluation system document retrieval models over-confidence dilemma