UniversalRAG: Retrieval-Augmented Generation over Multiple Corpora with
  Diverse Modalities and Granularities
Retrieval-Augmented Generation (RAG) has shown substantial promise in
improving factual accuracy by grounding model responses with external knowledge
relevant to queries. However, most existing RAG approaches are limited to a
text-only corpus, and while recent efforts have extended RAG to other
modalities such as images and videos, they typically operate over a single
modality-specific corpus. In contrast, real-world queries vary widely in the
type of knowledge they require, which a single type of knowledge source cannot
address. To address this, we introduce UniversalRAG, a novel RAG framework
designed to retrieve and integrate knowledge from heterogeneous sources with
diverse modalities and granularities. Specifically, motivated by the
observation that forcing all modalities into a unified representation space
derived from a single combined corpus causes a modality gap, where the
retrieval tends to favor items from the same modality as the query, we propose
a modality-aware routing mechanism that dynamically identifies the most
appropriate modality-specific corpus and performs targeted retrieval within it.
Also, beyond modality, we organize each modality into multiple granularity
levels, enabling fine-tuned retrieval tailored to the complexity and scope of
the query. We validate UniversalRAG on 8 benchmarks spanning multiple
modalities, showing its superiority over modality-specific and unified
baselines.
6811966ae20ba7d0683b8adc
Woongyeong Yeo
claimed_verified
2025-04-30T07:56:03.853Z
66d30f5fad293ffc4b7672bc
/avatars/6f164d813b947940a088820f8fd4dbe8.svg
Woongyeong Yeo
user
wgcyeo
6811966ae20ba7d0683b8add
Kangsan Kim
claimed_verified
2025-04-30T07:56:00.948Z
66ed7737f2f27a5dfd81ef09
/avatars/f45eea356e92ac7b3db23c2c92dec9fa.svg
Kangsan Kim
user
KangsanKim71
6811966ae20ba7d0683b8ade
Soyeong Jeong
6811966ae20ba7d0683b8adf
Jinheon Baek
claimed_verified
2025-04-30T07:56:07.351Z
63036b6c5c70c21d0ea79d48
/avatars/a7eb03f5cbd4eaa09fe807bbed8bc0f7.svg
Jinheon Baek
user
jinheon
6811966ae20ba7d0683b8ae0
Sung Ju Hwang
UniversalRAG: Retrieval-Augmented Generation over Multiple Corpora with
  Diverse Modalities and Granularities
Retrieval-Augmented Generation (RAG) has shown substantial promise in
improving factual accuracy by grounding model responses with external knowledge
relevant to queries. However, most existing RAG approaches are limited to a
text-only corpus, and while recent efforts have extended RAG to other
modalities such as images and videos, they typically operate over a single
modality-specific corpus. In contrast, real-world queries vary widely in the
type of knowledge they require, which a single type of knowledge source cannot
address. To address this, we introduce UniversalRAG, a novel RAG framework
designed to retrieve and integrate knowledge from heterogeneous sources with
diverse modalities and granularities. Specifically, motivated by the
observation that forcing all modalities into a unified representation space
derived from a single combined corpus causes a modality gap, where the
retrieval tends to favor items from the same modality as the query, we propose
a modality-aware routing mechanism that dynamically identifies the most
appropriate modality-specific corpus and performs targeted retrieval within it.
Also, beyond modality, we organize each modality into multiple granularity
levels, enabling fine-tuned retrieval tailored to the complexity and scope of
the query. We validate UniversalRAG on 8 benchmarks spanning multiple
modalities, showing its superiority over modality-specific and unified
baselines.
UniversalRAG enhances Retrieval-Augmented Generation by dynamically selecting and retrieving knowledge from various heterogeneous and modality-specific sources to improve factual accuracy across different types of queries.
Retrieval-Augmented Generation RAG factual accuracy external knowledge model responses queries modalities images videos UniversalRAG modality-aware routing mechanism granularities modality gap retrieval baselines