NaturalBench: Evaluating Vision-Language Models on Natural Adversarial
  Samples
Vision-language models (VLMs) have made significant progress in recent
visual-question-answering (VQA) benchmarks that evaluate complex
visio-linguistic reasoning. However, are these models truly effective? In this
work, we show that VLMs still struggle with natural images and questions that
humans can easily answer, which we term natural adversarial samples. We also
find it surprisingly easy to generate these VQA samples from natural image-text
corpora using off-the-shelf models like CLIP and ChatGPT. We propose a
semi-automated approach to collect a new benchmark, NaturalBench, for reliably
evaluating VLMs with 10,000 human-verified VQA samples. Crucially, we adopt a
vision-centric design by pairing each question with two images that
yield different answers, preventing blind solutions from answering without
using the images. This makes NaturalBench more challenging than previous
benchmarks that can be solved with commonsense priors. We evaluate 53
state-of-the-art VLMs on NaturalBench, showing that models like
LLaVA-OneVision, Cambrian-1, Llama3.2-Vision, Molmo, Qwen2-VL, and even GPT-4o
lag 50%-70% behind human performance (over 90%). We analyze why NaturalBench is
hard from two angles: (1) Compositionality: Solving NaturalBench requires
diverse visio-linguistic skills, including understanding attribute bindings,
object relationships, and advanced reasoning like logic and counting. To this
end, unlike prior work that uses a single tag per sample, we tag each
NaturalBench sample with 1 to 8 skill tags for fine-grained evaluation. (2)
Biases: NaturalBench exposes severe biases in VLMs, as models often choose the
same answer regardless of the image. Lastly, we apply our benchmark curation
method to diverse data sources, including long captions (over 100 words) and
non-English languages like Chinese and Hindi, highlighting its potential for
dynamic evaluations of VLMs.
6715c4518b7afb7724fa6aae
Baiqi Li
claimed_verified
2024-10-21T07:40:56.291Z
65ea393b1032135623fe8037
/avatars/cb15fa13076eb75e3e09206dafaf3dc7.svg
BaiqiLi
user
BaiqiL
6715c4518b7afb7724fa6aaf
Zhiqiu Lin
admin_assigned
2024-10-21T12:06:52.741Z
64c170190bfb901b04399295
/avatars/c30ce7566ae3497ddc989ec8918d37cc.svg
Zhiqiu Lin
user
zhiqiulin
6715c4518b7afb7724fa6ab0
Wenxuan Peng
admin_assigned
2024-10-21T12:07:22.881Z
63b91e47f270ad02f613237b
https://cdn-avatars.huggingface.co/v1/production/uploads/1673076442939-63b91e47f270ad02f613237b.jpeg
WENXUAN PENG
user
Lilymelon7
6715c4518b7afb7724fa6ab1
Jean de Dieu Nyandwi
claimed_verified
2024-10-21T11:42:11.775Z
62989cd743e990340bead77c
https://cdn-avatars.huggingface.co/v1/production/uploads/1654168730944-noauth.png
Jean de Dieu Nyandwi
user
Nyandwi
6715c4518b7afb7724fa6ab2
Daniel Jiang
claimed_verified
2024-10-21T20:01:33.369Z
66277092cee7ea5738cd04b6
/avatars/7894d5b8260dc426c89ffa237c6074e9.svg
Daniel Jiang
user
djiang04
6715c4518b7afb7724fa6ab3
Zixian Ma
admin_assigned
2024-10-21T12:07:32.845Z
6442cce0e255a338677e59ff
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VheGtt37p7aYIodlt9N3n.jpeg
Zixian Ma
user
zixianma
6715c4518b7afb7724fa6ab4
Simran Khanuja
admin_assigned
2024-10-21T12:08:00.335Z
63a4d079658851481f7e4394
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4d079658851481f7e4394/yptj_hJiaqHvEaqxLs0jT.jpeg
Simran Khanuja
user
skhanuja
6715c4518b7afb7724fa6ab5
Ranjay Krishna
admin_assigned
2024-10-21T12:08:07.227Z
66429868ab89e3a3a85668b0
/avatars/170e0daa454838deee2bf946f7118651.svg
Ranjay Krishna
user
ranjaykrishna
6715c4518b7afb7724fa6ab6
Graham Neubig
admin_assigned
2024-10-21T12:08:13.199Z
60de14638bedd2315529d43f
https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
Graham Neubig
user
gneubig
6715c4518b7afb7724fa6ab7
Deva Ramanan
admin_assigned
2024-10-21T12:08:21.627Z
6337151b0267ebcf02640eb6
/avatars/14a723cafc5587043bdfb19304fc202d.svg
Deva Ramanan
user
devakramanan
NaturalBench: Evaluating Vision-Language Models on Natural Adversarial
  Samples
Vision-language models (VLMs) have made significant progress in recent
visual-question-answering (VQA) benchmarks that evaluate complex
visio-linguistic reasoning. However, are these models truly effective? In this
work, we show that VLMs still struggle with natural images and questions that
humans can easily answer, which we term natural adversarial samples. We also
find it surprisingly easy to generate these VQA samples from natural image-text
corpora using off-the-shelf models like CLIP and ChatGPT. We propose a
semi-automated approach to collect a new benchmark, NaturalBench, for reliably
evaluating VLMs with 10,000 human-verified VQA samples. Crucially, we adopt a
vision-centric design by pairing each question with two images that
yield different answers, preventing blind solutions from answering without
using the images. This makes NaturalBench more challenging than previous
benchmarks that can be solved with commonsense priors. We evaluate 53
state-of-the-art VLMs on NaturalBench, showing that models like
LLaVA-OneVision, Cambrian-1, Llama3.2-Vision, Molmo, Qwen2-VL, and even GPT-4o
lag 50%-70% behind human performance (over 90%). We analyze why NaturalBench is
hard from two angles: (1) Compositionality: Solving NaturalBench requires
diverse visio-linguistic skills, including understanding attribute bindings,
object relationships, and advanced reasoning like logic and counting. To this
end, unlike prior work that uses a single tag per sample, we tag each
NaturalBench sample with 1 to 8 skill tags for fine-grained evaluation. (2)
Biases: NaturalBench exposes severe biases in VLMs, as models often choose the
same answer regardless of the image. Lastly, we apply our benchmark curation
method to diverse data sources, including long captions (over 100 words) and
non-English languages like Chinese and Hindi, highlighting its potential for
dynamic evaluations of VLMs.
VLMs perform poorly on a new benchmark, NaturalBench, which uses natural images and questions, highlighting challenges in understanding attribute bindings, object relationships, and advanced reasoning.
vision-language models VLMs visual-question-answering VQA NaturalBench vision-centric LLaVA-OneVision Cambrian-1 Llama3.2-Vision Molmo Qwen2-VL GPT-4o compositionality attribute bindings object relationships skill tags biases long captions non-English languages