UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG
Multimodal retrieval-augmented generation (MM-RAG) is a key approach for
applying large language models (LLMs) and agents to real-world knowledge bases,
yet current evaluations are fragmented, focusing on either text or images in
isolation or on simplified multimodal setups that fail to capture
document-centric multimodal use cases. In this paper, we introduce
UniDoc-Bench, the first large-scale, realistic benchmark for MM-RAG built from
70k real-world PDF pages across eight domains. Our pipeline extracts and links
evidence from text, tables, and figures, then generates 1,600 multimodal QA
pairs spanning factual retrieval, comparison, summarization, and logical
reasoning queries. To ensure reliability, 20% of QA pairs are validated by
multiple annotators and expert adjudication. UniDoc-Bench supports
apples-to-apples comparison across four paradigms: (1) text-only, (2)
image-only, (3) multimodal text-image fusion, and (4) multimodal joint
retrieval -- under a unified protocol with standardized candidate pools,
prompts, and evaluation metrics. Our experiments show that multimodal
text-image fusion RAG systems consistently outperform both unimodal and jointly
multimodal embedding-based retrieval, indicating that neither text nor images
alone are sufficient and that current multimodal embeddings remain inadequate.
Beyond benchmarking, our analysis reveals when and how visual context
complements textual evidence, uncovers systematic failure modes, and offers
actionable guidance for developing more robust MM-RAG pipelines.
68e7fb7e93680d7965e7c877
Xiangyu Peng
68e7fb7e93680d7965e7c878
Cab Qin
claimed_verified
2025-10-10T04:33:36.650Z
66a9bdfc35c89692442ba4b7
/avatars/ad87d1d2d81775eeb0920bf3ebe08cc2.svg
Can Qin
user
canqin001
68e7fb7e93680d7965e7c879
Zeyuan Chen
68e7fb7e93680d7965e7c87a
Ran Xu
68e7fb7e93680d7965e7c87b
Caiming Xiong
68e7fb7e93680d7965e7c87c
Chien-Sheng Wu
UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG
Multimodal retrieval-augmented generation (MM-RAG) is a key approach for
applying large language models (LLMs) and agents to real-world knowledge bases,
yet current evaluations are fragmented, focusing on either text or images in
isolation or on simplified multimodal setups that fail to capture
document-centric multimodal use cases. In this paper, we introduce
UniDoc-Bench, the first large-scale, realistic benchmark for MM-RAG built from
70k real-world PDF pages across eight domains. Our pipeline extracts and links
evidence from text, tables, and figures, then generates 1,600 multimodal QA
pairs spanning factual retrieval, comparison, summarization, and logical
reasoning queries. To ensure reliability, 20% of QA pairs are validated by
multiple annotators and expert adjudication. UniDoc-Bench supports
apples-to-apples comparison across four paradigms: (1) text-only, (2)
image-only, (3) multimodal text-image fusion, and (4) multimodal joint
retrieval -- under a unified protocol with standardized candidate pools,
prompts, and evaluation metrics. Our experiments show that multimodal
text-image fusion RAG systems consistently outperform both unimodal and jointly
multimodal embedding-based retrieval, indicating that neither text nor images
alone are sufficient and that current multimodal embeddings remain inadequate.
Beyond benchmarking, our analysis reveals when and how visual context
complements textual evidence, uncovers systematic failure modes, and offers
actionable guidance for developing more robust MM-RAG pipelines.
UniDoc-Bench is a large-scale benchmark for multimodal retrieval-augmented generation, evaluating systems across text, images, and their fusion in real-world document-centric scenarios.
multimodal retrieval-augmented generation large language models knowledge bases UniDoc-Bench PDF pages multimodal QA pairs factual retrieval comparison summarization logical reasoning text-only image-only multimodal text-image fusion multimodal joint retrieval candidate pools prompts evaluation metrics visual context textual evidence failure modes robust MM-RAG pipelines