VideoRAG: Retrieval-Augmented Generation over Video Corpus
Retrieval-Augmented Generation (RAG) is a powerful strategy to address the
issue of generating factually incorrect outputs in foundation models by
retrieving external knowledge relevant to queries and incorporating it into
their generation process. However, existing RAG approaches have primarily
focused on textual information, with some recent advancements beginning to
consider images, and they largely overlook videos, a rich source of multimodal
knowledge capable of representing events, processes, and contextual details
more effectively than any other modality. While a few recent studies explore
the integration of videos in the response generation process, they either
predefine query-associated videos without retrieving them according to queries,
or convert videos into the textual descriptions without harnessing their
multimodal richness. To tackle these, we introduce VideoRAG, a novel framework
that not only dynamically retrieves relevant videos based on their relevance
with queries but also utilizes both visual and textual information of videos in
the output generation. Further, to operationalize this, our method revolves
around the recent advance of Large Video Language Models (LVLMs), which enable
the direct processing of video content to represent it for retrieval and
seamless integration of the retrieved videos jointly with queries. We
experimentally validate the effectiveness of VideoRAG, showcasing that it is
superior to relevant baselines.
678474cb65495e1686093192
Soyeong Jeong
claimed_verified
2025-01-13T09:18:35.528Z
64e5a1cd4c20016ec9020ec8
/avatars/d7ffe7fbbe39c0a013375357457c57b3.svg
Soyeong
user
starsuzi
678474cb65495e1686093193
Kangsan Kim
claimed_verified
2025-01-13T08:54:32.431Z
66ed7737f2f27a5dfd81ef09
/avatars/f45eea356e92ac7b3db23c2c92dec9fa.svg
Kangsan Kim
user
KangsanKim71
678474cb65495e1686093194
Jinheon Baek
admin_assigned
2025-01-13T09:15:25.765Z
63036b6c5c70c21d0ea79d48
/avatars/a7eb03f5cbd4eaa09fe807bbed8bc0f7.svg
Jinheon Baek
user
jinheon
678474cb65495e1686093195
Sung Ju Hwang
VideoRAG: Retrieval-Augmented Generation over Video Corpus
Retrieval-Augmented Generation (RAG) is a powerful strategy to address the
issue of generating factually incorrect outputs in foundation models by
retrieving external knowledge relevant to queries and incorporating it into
their generation process. However, existing RAG approaches have primarily
focused on textual information, with some recent advancements beginning to
consider images, and they largely overlook videos, a rich source of multimodal
knowledge capable of representing events, processes, and contextual details
more effectively than any other modality. While a few recent studies explore
the integration of videos in the response generation process, they either
predefine query-associated videos without retrieving them according to queries,
or convert videos into the textual descriptions without harnessing their
multimodal richness. To tackle these, we introduce VideoRAG, a novel framework
that not only dynamically retrieves relevant videos based on their relevance
with queries but also utilizes both visual and textual information of videos in
the output generation. Further, to operationalize this, our method revolves
around the recent advance of Large Video Language Models (LVLMs), which enable
the direct processing of video content to represent it for retrieval and
seamless integration of the retrieved videos jointly with queries. We
experimentally validate the effectiveness of VideoRAG, showcasing that it is
superior to relevant baselines.
VideoRAG dynamically retrieves relevant videos based on queries and utilizes both visual and textual information for output generation, outperforming existing methods in response accuracy.
Retrieval-Augmented Generation RAG Large Video Language Models LVLMs video retrieval visual information multimodal knowledge video processing