How Far Are We from Intelligent Visual Deductive Reasoning?
Vision-Language Models (VLMs) such as GPT-4V have recently demonstrated
incredible strides on diverse vision language tasks. We dig into vision-based
deductive reasoning, a more sophisticated but less explored realm, and find
previously unexposed blindspots in the current SOTA VLMs. Specifically, we
leverage Raven's Progressive Matrices (RPMs), to assess VLMs' abilities to
perform multi-hop relational and deductive reasoning relying solely on visual
clues. We perform comprehensive evaluations of several popular VLMs employing
standard strategies such as in-context learning, self-consistency, and
Chain-of-thoughts (CoT) on three diverse datasets, including the Mensa IQ test,
IntelligenceTest, and RAVEN. The results reveal that despite the impressive
capabilities of LLMs in text-based reasoning, we are still far from achieving
comparable proficiency in visual deductive reasoning. We found that certain
standard strategies that are effective when applied to LLMs do not seamlessly
translate to the challenges presented by visual reasoning tasks. Moreover, a
detailed analysis reveals that VLMs struggle to solve these tasks mainly
because they are unable to perceive and comprehend multiple, confounding
abstract patterns in RPM examples.
65ea7a22859bf22ab4521798
Yizhe Zhang
admin_assigned
2024-03-08T09:46:24.870Z
5e64858c87403103f9f1055d
https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/96-oNO2OLVkAFOojMaV0T.jpeg
Yizhe Zhang
user
yizhezhang
65ea7a22859bf22ab4521799
He Bai
claimed_verified
2024-03-08T08:33:17.432Z
5f024bd8ff69163f6f59e775
https://cdn-avatars.huggingface.co/v1/production/uploads/1652111948107-5f024bd8ff69163f6f59e775.png
Richard He Bai
user
richardbaihe
65ea7a22859bf22ab452179a
Ruixiang Zhang
65ea7a22859bf22ab452179b
Jiatao Gu
admin_assigned
2024-03-08T09:47:22.141Z
6164e72d73996c363c52e66d
https://cdn-avatars.huggingface.co/v1/production/uploads/1634002684894-noauth.png
Jiatao Gu
user
thomagram
65ea7a22859bf22ab452179c
Shuangfei Zhai
claimed_verified
2024-12-13T15:46:05.132Z
675bf04240726aac79ae8e7b
/avatars/8ae95255dfce91a6f5e76f14eae9e9e5.svg
Shuangfei Zhai
user
shuangfei
65ea7a22859bf22ab452179d
Josh Susskind
claimed_verified
2024-11-22T16:30:49.961Z
6470d2247fd7ecdbd0ec3cc9
/avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
Joshua M. Susskind
user
jsusskind
65ea7a22859bf22ab452179e
Navdeep Jaitly
admin_assigned
2024-03-08T09:47:40.093Z
631119df88942700629f4aab
/avatars/251c20a8fd48831a8c0cc0d198a42455.svg
Navdeep Jaitly
user
ndjaitly
How Far Are We from Intelligent Visual Deductive Reasoning?
Vision-Language Models (VLMs) such as GPT-4V have recently demonstrated
incredible strides on diverse vision language tasks. We dig into vision-based
deductive reasoning, a more sophisticated but less explored realm, and find
previously unexposed blindspots in the current SOTA VLMs. Specifically, we
leverage Raven's Progressive Matrices (RPMs), to assess VLMs' abilities to
perform multi-hop relational and deductive reasoning relying solely on visual
clues. We perform comprehensive evaluations of several popular VLMs employing
standard strategies such as in-context learning, self-consistency, and
Chain-of-thoughts (CoT) on three diverse datasets, including the Mensa IQ test,
IntelligenceTest, and RAVEN. The results reveal that despite the impressive
capabilities of LLMs in text-based reasoning, we are still far from achieving
comparable proficiency in visual deductive reasoning. We found that certain
standard strategies that are effective when applied to LLMs do not seamlessly
translate to the challenges presented by visual reasoning tasks. Moreover, a
detailed analysis reveals that VLMs struggle to solve these tasks mainly
because they are unable to perceive and comprehend multiple, confounding
abstract patterns in RPM examples.
VLMs show deficiencies in visual deductive reasoning, particularly in multi-hop relational tasks, despite strong performance in text-based reasoning.
Vision-Language Models VLMs GPT-4V Raven's Progressive Matrices RPMs in-context learning self-consistency Chain-of-thoughts CoT visual deductive reasoning multi-hop relational reasoning Mensa IQ test IntelligenceTest RAVEN