Guiding Vision-Language Model Selection for Visual Question-Answering
  Across Tasks, Domains, and Knowledge Types
Visual Question-Answering (VQA) has become a key use-case in several
applications to aid user experience, particularly after Vision-Language Models
(VLMs) achieving good results in zero-shot inference. But evaluating different
VLMs for an application requirement using a standardized framework in practical
settings is still challenging. This paper introduces a comprehensive framework
for evaluating VLMs tailored to VQA tasks in practical settings. We present a
novel dataset derived from established VQA benchmarks, annotated with task
types, application domains, and knowledge types, three key practical aspects on
which tasks can vary. We also introduce GoEval, a multimodal evaluation metric
developed using GPT-4o, achieving a correlation factor of 56.71% with human
judgments. Our experiments with ten state-of-the-art VLMs reveals that no
single model excelling universally, making appropriate selection a key design
decision. Proprietary models such as Gemini-1.5-Pro and GPT-4o-mini generally
outperform others, though open-source models like InternVL-2-8B and
CogVLM-2-Llama-3-19B demonstrate competitive strengths in specific contexts,
while providing additional advantages. This study guides the selection of VLMs
based on specific task requirements and resource constraints, and can also be
extended to other vision-language tasks.
66e9108fcfd2f509914b1e94
Neelabh Sinha
admin_assigned
2024-09-17T07:45:52.171Z
6639a17d7256327c68a0841e
/avatars/28fcd935149c2d89661c6f642842c310.svg
Neelabh Sinha
user
neelabhsinha
66e9108fcfd2f509914b1e95
Vinija Jain
66e9108fcfd2f509914b1e96
Aman Chadha
claimed_verified
2024-09-17T07:11:18.674Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
user
amanchadha
Guiding Vision-Language Model Selection for Visual Question-Answering
  Across Tasks, Domains, and Knowledge Types
Visual Question-Answering (VQA) has become a key use-case in several
applications to aid user experience, particularly after Vision-Language Models
(VLMs) achieving good results in zero-shot inference. But evaluating different
VLMs for an application requirement using a standardized framework in practical
settings is still challenging. This paper introduces a comprehensive framework
for evaluating VLMs tailored to VQA tasks in practical settings. We present a
novel dataset derived from established VQA benchmarks, annotated with task
types, application domains, and knowledge types, three key practical aspects on
which tasks can vary. We also introduce GoEval, a multimodal evaluation metric
developed using GPT-4o, achieving a correlation factor of 56.71% with human
judgments. Our experiments with ten state-of-the-art VLMs reveals that no
single model excelling universally, making appropriate selection a key design
decision. Proprietary models such as Gemini-1.5-Pro and GPT-4o-mini generally
outperform others, though open-source models like InternVL-2-8B and
CogVLM-2-Llama-3-19B demonstrate competitive strengths in specific contexts,
while providing additional advantages. This study guides the selection of VLMs
based on specific task requirements and resource constraints, and can also be
extended to other vision-language tasks.
A framework for evaluating Vision-Language Models in VQA tasks using a novel dataset and GoEval metric, highlighting performance variation across models and guiding selection based on task and resource constraints.
Vision-Language Models VQA zero-shot inference evaluation framework multimodal evaluation metric GoEval GPT-4o task types application domains knowledge types Gemini-1.5-Pro GPT-4o-mini InternVL-2-8B CogVLM-2-Llama-3-19B