Unsolvable Problem Detection: Evaluating Trustworthiness of Vision
  Language Models
This paper introduces a novel and significant challenge for Vision Language
Models (VLMs), termed Unsolvable Problem Detection (UPD). UPD examines the
VLM's ability to withhold answers when faced with unsolvable problems in the
context of Visual Question Answering (VQA) tasks. UPD encompasses three
distinct settings: Absent Answer Detection (AAD), Incompatible Answer Set
Detection (IASD), and Incompatible Visual Question Detection (IVQD). To deeply
investigate the UPD problem, extensive experiments indicate that most VLMs,
including GPT-4V and LLaVA-Next-34B, struggle with our benchmarks to varying
extents, highlighting significant room for the improvements. To address UPD, we
explore both training-free and training-based solutions, offering new insights
into their effectiveness and limitations. We hope our insights, together with
future efforts within the proposed UPD settings, will enhance the broader
understanding and development of more practical and reliable VLMs.
660a1b750b7d615a56a13e19
Atsuyuki Miyai
claimed_verified
2024-04-11T05:51:49.719Z
6527b37c0ae663e384eb1b85
https://cdn-avatars.huggingface.co/v1/production/uploads/6527b37c0ae663e384eb1b85/zKWa8h6YU4BWfcitpM5Pl.png
Atsuyuki Miyai
user
AtsuMiyai
660a1b750b7d615a56a13e1a
Jingkang Yang
660a1b750b7d615a56a13e1b
Jingyang Zhang
claimed_verified
2024-04-05T10:04:29.626Z
63f7773ed36951307fd02d18
https://cdn-avatars.huggingface.co/v1/production/uploads/63f7773ed36951307fd02d18/l8KmKNYvBXbUDHaH9BWL6.jpeg
Jingyang Zhang
user
zjysteven
660a1b750b7d615a56a13e1c
Yifei Ming
claimed_verified
2024-10-11T07:46:20.294Z
641787ad1f1f3b0fa80f5340
/avatars/489b7c999bce2160e27c9b487e1026ef.svg
yifei ming
user
alvinming
660a1b750b7d615a56a13e1d
Qing Yu
660a1b750b7d615a56a13e1e
Go Irie
660a1b750b7d615a56a13e1f
Yixuan Li
660a1b750b7d615a56a13e20
Hai Li
660a1b750b7d615a56a13e21
Ziwei Liu
660a1b750b7d615a56a13e22
Kiyoharu Aizawa
Unsolvable Problem Detection: Evaluating Trustworthiness of Vision
  Language Models
This paper introduces a novel and significant challenge for Vision Language
Models (VLMs), termed Unsolvable Problem Detection (UPD). UPD examines the
VLM's ability to withhold answers when faced with unsolvable problems in the
context of Visual Question Answering (VQA) tasks. UPD encompasses three
distinct settings: Absent Answer Detection (AAD), Incompatible Answer Set
Detection (IASD), and Incompatible Visual Question Detection (IVQD). To deeply
investigate the UPD problem, extensive experiments indicate that most VLMs,
including GPT-4V and LLaVA-Next-34B, struggle with our benchmarks to varying
extents, highlighting significant room for the improvements. To address UPD, we
explore both training-free and training-based solutions, offering new insights
into their effectiveness and limitations. We hope our insights, together with
future efforts within the proposed UPD settings, will enhance the broader
understanding and development of more practical and reliable VLMs.
Unsolvable Problem Detection evaluates Vision Language Models' ability to identify unsolvable questions in Visual Question Answering through various settings, revealing areas for improvement and offering both training-free and training-based solutions.
Vision Language Models Unsolvable Problem Detection Visual Question Answering Absent Answer Detection Incompatible Answer Set Detection Incompatible Visual Question Detection GPT-4V LLaVA-Next-34B