Knowledge-based Visual Question Answer with Multimodal Processing,
  Retrieval and Filtering
Knowledge-based visual question answering (KB-VQA) requires visual language
models (VLMs) to integrate visual understanding with external knowledge
retrieval. Although retrieval-augmented generation (RAG) achieves significant
advances in this task by combining knowledge-base querying, it still struggles
with the quality of multimodal queries and the relevance of retrieved results.
To overcome these challenges, we propose a novel three-stage method, termed
Wiki-PRF, including Processing, Retrieval and Filtering stages. The processing
stage dynamically invokes visual tools to extract precise multimodal
information for retrieval. The retrieval stage integrates visual and text
features to achieve multimodal knowledge retrieval. The filtering stage
performs relevance filtering and concentration on retrieval results. To this
end, we introduce a visual language model trained with answer accuracy and
format consistency as reward signals via a reinforcement learning manner. This
enhances the model's reasoning, tool invocation for accurate queries, and
filtering of irrelevant content. Experiments on benchmark datasets (E-VQA and
InfoSeek) show significant improvements~(36.0 and 42.8) in answer quality,
achieving state-of-the-art performance. Code is available at
https://github.com/cqu-student/Wiki-PRF
68f33e1b8589920bf4d31af3
Yuyang Hong
claimed_verified
2025-10-20T12:12:55.883Z
66e03d7123e5f162e7e5c5ac
/avatars/28089b6ffeacafff2c798b7579689c1e.svg
hongyuyang
user
hongyuyang23casia
68f33e1b8589920bf4d31af4
Jiaqi Gu
68f33e1b8589920bf4d31af5
Qi Yang
68f33e1b8589920bf4d31af6
Lubin Fan
68f33e1b8589920bf4d31af7
Yue Wu
68f33e1b8589920bf4d31af8
Ying Wang
68f33e1b8589920bf4d31af9
Kun Ding
68f33e1b8589920bf4d31afa
Shiming Xiang
68f33e1b8589920bf4d31afb
Jieping Ye
Knowledge-based Visual Question Answer with Multimodal Processing,
  Retrieval and Filtering
Knowledge-based visual question answering (KB-VQA) requires visual language
models (VLMs) to integrate visual understanding with external knowledge
retrieval. Although retrieval-augmented generation (RAG) achieves significant
advances in this task by combining knowledge-base querying, it still struggles
with the quality of multimodal queries and the relevance of retrieved results.
To overcome these challenges, we propose a novel three-stage method, termed
Wiki-PRF, including Processing, Retrieval and Filtering stages. The processing
stage dynamically invokes visual tools to extract precise multimodal
information for retrieval. The retrieval stage integrates visual and text
features to achieve multimodal knowledge retrieval. The filtering stage
performs relevance filtering and concentration on retrieval results. To this
end, we introduce a visual language model trained with answer accuracy and
format consistency as reward signals via a reinforcement learning manner. This
enhances the model's reasoning, tool invocation for accurate queries, and
filtering of irrelevant content. Experiments on benchmark datasets (E-VQA and
InfoSeek) show significant improvements~(36.0 and 42.8) in answer quality,
achieving state-of-the-art performance. Code is available at
https://github.com/cqu-student/Wiki-PRF
A novel three-stage method, Wiki-PRF, enhances knowledge-based visual question answering by improving multimodal query quality and relevance through visual language models and reinforcement learning.
knowledge-based visual question answering KB-VQA visual language models VLMs retrieval-augmented generation RAG multimodal queries multimodal knowledge retrieval relevance filtering reinforcement learning answer accuracy format consistency E-VQA InfoSeek