ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic
  Iterative Reasoning Agents
Understanding information from visually rich documents remains a significant
challenge for traditional Retrieval-Augmented Generation (RAG) methods.
Existing benchmarks predominantly focus on image-based question answering (QA),
overlooking the fundamental challenges of efficient retrieval, comprehension,
and reasoning within dense visual documents. To bridge this gap, we introduce
ViDoSeek, a novel dataset designed to evaluate RAG performance on visually rich
documents requiring complex reasoning. Based on it, we identify key limitations
in current RAG approaches: (i) purely visual retrieval methods struggle to
effectively integrate both textual and visual features, and (ii) previous
approaches often allocate insufficient reasoning tokens, limiting their
effectiveness. To address these challenges, we propose ViDoRAG, a novel
multi-agent RAG framework tailored for complex reasoning across visual
documents. ViDoRAG employs a Gaussian Mixture Model (GMM)-based hybrid strategy
to effectively handle multi-modal retrieval. To further elicit the model's
reasoning capabilities, we introduce an iterative agent workflow incorporating
exploration, summarization, and reflection, providing a framework for
investigating test-time scaling in RAG domains. Extensive experiments on
ViDoSeek validate the effectiveness and generalization of our approach.
Notably, ViDoRAG outperforms existing methods by over 10% on the competitive
ViDoSeek benchmark.
67bef5a6070ec160042d99f4
Qiuchen Wang
claimed_verified
2025-02-28T12:15:57.850Z
657429d833e5a4bf5b278615
/avatars/ed7e28c1b9a7bed1cad864c992cdcc69.svg
QiuchenWang
user
autumncc
67bef5a6070ec160042d99f5
Ruixue Ding
67bef5a6070ec160042d99f6
Zehui Chen
admin_assigned
2025-03-03T09:32:18.129Z
64892d31cbda0d1cdb956897
/avatars/3cdafe03a8295124636347d15a099aaf.svg
Zehui Chen
user
lovesnowbest
67bef5a6070ec160042d99f7
Weiqi Wu
admin_assigned
2025-03-03T09:32:12.075Z
65351cbe6141b3927afaed17
/avatars/5abf5f2c4ab329e63a7f45c15c9dfb93.svg
weiqi wu
user
vickywu
67bef5a6070ec160042d99f8
Shihang Wang
admin_assigned
2025-03-03T09:32:05.679Z
62e8efb14210d3fe69eacb42
/avatars/2feadd75274bf353b910f4679ef72b39.svg
Shihang Wang
user
shihang
67bef5a6070ec160042d99f9
Pengjun Xie
admin_assigned
2025-03-03T09:31:59.813Z
63a091e42fabbbb89991f5ce
/avatars/d55485b06461764c36c9edf9d6e8892c.svg
pengjun xie
user
xpjandy
67bef5a6070ec160042d99fa
Feng Zhao
ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic
  Iterative Reasoning Agents
Understanding information from visually rich documents remains a significant
challenge for traditional Retrieval-Augmented Generation (RAG) methods.
Existing benchmarks predominantly focus on image-based question answering (QA),
overlooking the fundamental challenges of efficient retrieval, comprehension,
and reasoning within dense visual documents. To bridge this gap, we introduce
ViDoSeek, a novel dataset designed to evaluate RAG performance on visually rich
documents requiring complex reasoning. Based on it, we identify key limitations
in current RAG approaches: (i) purely visual retrieval methods struggle to
effectively integrate both textual and visual features, and (ii) previous
approaches often allocate insufficient reasoning tokens, limiting their
effectiveness. To address these challenges, we propose ViDoRAG, a novel
multi-agent RAG framework tailored for complex reasoning across visual
documents. ViDoRAG employs a Gaussian Mixture Model (GMM)-based hybrid strategy
to effectively handle multi-modal retrieval. To further elicit the model's
reasoning capabilities, we introduce an iterative agent workflow incorporating
exploration, summarization, and reflection, providing a framework for
investigating test-time scaling in RAG domains. Extensive experiments on
ViDoSeek validate the effectiveness and generalization of our approach.
Notably, ViDoRAG outperforms existing methods by over 10% on the competitive
ViDoSeek benchmark.
ViDoRAG, a multi-agent RAG framework using GMM-based hybrid strategy, enhances performance on visually rich documents requiring complex reasoning by addressing limitations in feature integration and reasoning capacity.
Retrieval-Augmented Generation (RAG) ViDoSeek Gaussian Mixture Model (GMM) multi-modal retrieval iterative agent workflow exploration summarization reflection test-time scaling reasoning tokens