Vision Search Assistant: Empower Vision-Language Models as Multimodal
  Search Engines
Search engines enable the retrieval of unknown information with texts.
However, traditional methods fall short when it comes to understanding
unfamiliar visual content, such as identifying an object that the model has
never seen before. This challenge is particularly pronounced for large
vision-language models (VLMs): if the model has not been exposed to the object
depicted in an image, it struggles to generate reliable answers to the user's
question regarding that image. Moreover, as new objects and events continuously
emerge, frequently updating VLMs is impractical due to heavy computational
burdens. To address this limitation, we propose Vision Search Assistant, a
novel framework that facilitates collaboration between VLMs and web agents.
This approach leverages VLMs' visual understanding capabilities and web agents'
real-time information access to perform open-world Retrieval-Augmented
Generation via the web. By integrating visual and textual representations
through this collaboration, the model can provide informed responses even when
the image is novel to the system. Extensive experiments conducted on both
open-set and closed-set QA benchmarks demonstrate that the Vision Search
Assistant significantly outperforms the other models and can be widely applied
to existing VLMs.
67204b36cbdaf3111383d445
Zhixin Zhang
67204b36cbdaf3111383d446
Yiyuan Zhang
67204b36cbdaf3111383d447
Xiaohan Ding
67204b36cbdaf3111383d448
Xiangyu Yue
Vision Search Assistant: Empower Vision-Language Models as Multimodal
  Search Engines
Search engines enable the retrieval of unknown information with texts.
However, traditional methods fall short when it comes to understanding
unfamiliar visual content, such as identifying an object that the model has
never seen before. This challenge is particularly pronounced for large
vision-language models (VLMs): if the model has not been exposed to the object
depicted in an image, it struggles to generate reliable answers to the user's
question regarding that image. Moreover, as new objects and events continuously
emerge, frequently updating VLMs is impractical due to heavy computational
burdens. To address this limitation, we propose Vision Search Assistant, a
novel framework that facilitates collaboration between VLMs and web agents.
This approach leverages VLMs' visual understanding capabilities and web agents'
real-time information access to perform open-world Retrieval-Augmented
Generation via the web. By integrating visual and textual representations
through this collaboration, the model can provide informed responses even when
the image is novel to the system. Extensive experiments conducted on both
open-set and closed-set QA benchmarks demonstrate that the Vision Search
Assistant significantly outperforms the other models and can be widely applied
to existing VLMs.
Vision Search Assistant enhances VLMs by collaborating with web agents for retrieving and generating information about unseen visual content, improving performance on image-based QA tasks.
vision-language models VLMs visual understanding web agents open-world Retrieval-Augmented Generation visual and textual representations open-set QA closed-set QA