VLM^2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit
  Matching Visual Cues
Visually linking matching cues is a crucial ability in daily life, such as
identifying the same person in multiple photos based on their cues, even
without knowing who they are. Despite the extensive knowledge that
vision-language models (VLMs) possess, it remains largely unexplored whether
they are capable of performing this fundamental task. To address this, we
introduce VLM^2-Bench, a benchmark designed to assess whether VLMs can
Visually Link Matching cues, with 9 subtasks and over 3,000 test cases.
Comprehensive evaluation across eight open-source VLMs and GPT-4o, along with
further analysis of various language-side and vision-side prompting methods,
leads to a total of eight key findings. We identify critical challenges in
models' ability to link visual cues, highlighting a significant performance gap
where even GPT-4o lags 34.80% behind humans. Based on these insights, we
advocate for (i) enhancing core visual capabilities to improve adaptability and
reduce reliance on prior knowledge, (ii) establishing clearer principles for
integrating language-based reasoning in vision-centric tasks to prevent
unnecessary biases, and (iii) shifting vision-text training paradigms toward
fostering models' ability to independently structure and infer relationships
among visual cues.
67b8922ef6632327952ec1e1
Jianshu Zhang
extracted_infirmed
2025-06-03T01:59:17.338Z
67b8922ef6632327952ec1e2
Dongyu Yao
claimed_verified
2025-02-24T09:20:43.528Z
64b0377121a001042bc0d274
https://cdn-avatars.huggingface.co/v1/production/uploads/64b0377121a001042bc0d274/Hk8yI5_s7ey5o9SVZzXrB.png
Dongyu Yao
user
RainJamesY
67b8922ef6632327952ec1e3
Renjie Pi
admin_assigned
2025-02-24T13:18:30.254Z
63f45b8d520c14618930d175
/avatars/a20994594579b52a8be8bd2c4acbb913.svg
renjie
user
renjiepi
67b8922ef6632327952ec1e4
Paul Pu Liang
67b8922ef6632327952ec1e5
Yi R.
67b8922ef6632327952ec1e6
Fung
extracted_pending
2025-02-26T01:33:46.195Z
67be6f4f22513d7d52d7ef66
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/jQEVQufb6BOb9cQpb6NIt.png
San Yi
user
Sanyia
VLM^2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit
  Matching Visual Cues
Visually linking matching cues is a crucial ability in daily life, such as
identifying the same person in multiple photos based on their cues, even
without knowing who they are. Despite the extensive knowledge that
vision-language models (VLMs) possess, it remains largely unexplored whether
they are capable of performing this fundamental task. To address this, we
introduce VLM^2-Bench, a benchmark designed to assess whether VLMs can
Visually Link Matching cues, with 9 subtasks and over 3,000 test cases.
Comprehensive evaluation across eight open-source VLMs and GPT-4o, along with
further analysis of various language-side and vision-side prompting methods,
leads to a total of eight key findings. We identify critical challenges in
models' ability to link visual cues, highlighting a significant performance gap
where even GPT-4o lags 34.80% behind humans. Based on these insights, we
advocate for (i) enhancing core visual capabilities to improve adaptability and
reduce reliance on prior knowledge, (ii) establishing clearer principles for
integrating language-based reasoning in vision-centric tasks to prevent
unnecessary biases, and (iii) shifting vision-text training paradigms toward
fostering models' ability to independently structure and infer relationships
among visual cues.
VLM$^2$-Bench evaluates vision-language models' ability to link visual cues across images, highlighting significant performance gaps and suggesting improvements to enhance visual capabilities and integration of language-based reasoning.
vision-language models VLM$^2$-Bench visual cues prompting methods visual capabilities language-based reasoning vision-centric tasks