AVIS: Autonomous Visual Information Seeking with Large Language Models
In this paper, we propose an autonomous information seeking visual question
answering framework, AVIS. Our method leverages a Large Language Model (LLM) to
dynamically strategize the utilization of external tools and to investigate
their outputs, thereby acquiring the indispensable knowledge needed to provide
answers to the posed questions. Responding to visual questions that necessitate
external knowledge, such as "What event is commemorated by the building
depicted in this image?", is a complex task. This task presents a combinatorial
search space that demands a sequence of actions, including invoking APIs,
analyzing their responses, and making informed decisions. We conduct a user
study to collect a variety of instances of human decision-making when faced
with this task. This data is then used to design a system comprised of three
components: an LLM-powered planner that dynamically determines which tool to
use next, an LLM-powered reasoner that analyzes and extracts key information
from the tool outputs, and a working memory component that retains the acquired
information throughout the process. The collected user behavior serves as a
guide for our system in two key ways. First, we create a transition graph by
analyzing the sequence of decisions made by users. This graph delineates
distinct states and confines the set of actions available at each state.
Second, we use examples of user decision-making to provide our LLM-powered
planner and reasoner with relevant contextual instances, enhancing their
capacity to make informed decisions. We show that AVIS achieves
state-of-the-art results on knowledge-intensive visual question answering
benchmarks such as Infoseek and OK-VQA.
648bc615d0f0f5b02eaf5d95
Ziniu Hu
648bc615d0f0f5b02eaf5d96
Ahmet Iscen
admin_assigned
2023-06-16T16:14:37.813Z
64885bd4dfa0998196b1cf0a
/avatars/c3e2bbe793cdd146aea27c0011aace71.svg
Ahmet Iscen
user
ahmetius
648bc615d0f0f5b02eaf5d97
Chen Sun
648bc615d0f0f5b02eaf5d98
Kai-Wei Chang
admin_assigned
2023-06-16T16:15:59.421Z
60b7b9d71b90c5d07c23fbd0
https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
Kai-Wei Chang
user
kaiweichang
648bc615d0f0f5b02eaf5d99
Yizhou Sun
648bc615d0f0f5b02eaf5d9a
David A Ross
648bc615d0f0f5b02eaf5d9b
Cordelia Schmid
648bc615d0f0f5b02eaf5d9c
Alireza Fathi
admin_assigned
2023-06-16T16:16:51.620Z
645685170a1ea2229a28dbb2
/avatars/b83e043784b6db225ae4d0422aabbb3a.svg
alireza fathi
user
alireza007
AVIS: Autonomous Visual Information Seeking with Large Language Models
In this paper, we propose an autonomous information seeking visual question
answering framework, AVIS. Our method leverages a Large Language Model (LLM) to
dynamically strategize the utilization of external tools and to investigate
their outputs, thereby acquiring the indispensable knowledge needed to provide
answers to the posed questions. Responding to visual questions that necessitate
external knowledge, such as "What event is commemorated by the building
depicted in this image?", is a complex task. This task presents a combinatorial
search space that demands a sequence of actions, including invoking APIs,
analyzing their responses, and making informed decisions. We conduct a user
study to collect a variety of instances of human decision-making when faced
with this task. This data is then used to design a system comprised of three
components: an LLM-powered planner that dynamically determines which tool to
use next, an LLM-powered reasoner that analyzes and extracts key information
from the tool outputs, and a working memory component that retains the acquired
information throughout the process. The collected user behavior serves as a
guide for our system in two key ways. First, we create a transition graph by
analyzing the sequence of decisions made by users. This graph delineates
distinct states and confines the set of actions available at each state.
Second, we use examples of user decision-making to provide our LLM-powered
planner and reasoner with relevant contextual instances, enhancing their
capacity to make informed decisions. We show that AVIS achieves
state-of-the-art results on knowledge-intensive visual question answering
benchmarks such as Infoseek and OK-VQA.
AUTONOMOUS INFORMATION SEEKING VISUAL QUESTION ANSWERING FRAMEWORK (AVIS) leverages LLMs to strategize and utilize external tools, achieving state-of-the-art results in knowledge-intensive VQA.
Large Language Model (LLM) autonomous information seeking visual question answering external tools API invocation working memory transition graph contextual instances Infoseek OK-VQA