Can Large Multimodal Models Actively Recognize Faulty Inputs? A
  Systematic Evaluation Framework of Their Input Scrutiny Ability
Large Multimodal Models (LMMs) have witnessed remarkable growth, showcasing
formidable capabilities in handling intricate multimodal tasks with exceptional
performance. Recent research has underscored the inclination of large language
models to passively accept defective inputs, often resulting in futile
reasoning on invalid prompts. However, the same critical question of whether
LMMs can actively detect and scrutinize erroneous inputs still remains
unexplored. To address this gap, we introduce the Input Scrutiny Ability
Evaluation Framework (ISEval), which encompasses seven categories of flawed
premises and three evaluation metrics. Our extensive evaluation of ten advanced
LMMs has identified key findings. Most models struggle to actively detect
flawed textual premises without guidance, which reflects a strong reliance on
explicit prompts for premise error identification. Error type affects
performance: models excel at identifying logical fallacies but struggle with
surface-level linguistic errors and certain conditional flaws. Modality trust
varies-Gemini 2.5 pro and Claude Sonnet 4 balance visual and textual info,
while aya-vision-8b over-rely on text in conflicts. These insights underscore
the urgent need to enhance LMMs' proactive verification of input validity and
shed novel insights into mitigating the problem. The code is available at
https://github.com/MLGroupJLU/LMM_ISEval.
68957a1b48b0ae5ca2710de5
Haiqi Yang
68957a1b48b0ae5ca2710de6
Jinzhe Li
68957a1b48b0ae5ca2710de7
Gengxu Li
68957a1b48b0ae5ca2710de8
Yi Chang
68957a1b48b0ae5ca2710de9
Yuan Wu
Can Large Multimodal Models Actively Recognize Faulty Inputs? A
  Systematic Evaluation Framework of Their Input Scrutiny Ability
Large Multimodal Models (LMMs) have witnessed remarkable growth, showcasing
formidable capabilities in handling intricate multimodal tasks with exceptional
performance. Recent research has underscored the inclination of large language
models to passively accept defective inputs, often resulting in futile
reasoning on invalid prompts. However, the same critical question of whether
LMMs can actively detect and scrutinize erroneous inputs still remains
unexplored. To address this gap, we introduce the Input Scrutiny Ability
Evaluation Framework (ISEval), which encompasses seven categories of flawed
premises and three evaluation metrics. Our extensive evaluation of ten advanced
LMMs has identified key findings. Most models struggle to actively detect
flawed textual premises without guidance, which reflects a strong reliance on
explicit prompts for premise error identification. Error type affects
performance: models excel at identifying logical fallacies but struggle with
surface-level linguistic errors and certain conditional flaws. Modality trust
varies-Gemini 2.5 pro and Claude Sonnet 4 balance visual and textual info,
while aya-vision-8b over-rely on text in conflicts. These insights underscore
the urgent need to enhance LMMs' proactive verification of input validity and
shed novel insights into mitigating the problem. The code is available at
https://github.com/MLGroupJLU/LMM_ISEval.
ISEval framework evaluates large multimodal models' ability to detect flawed inputs, revealing challenges in identifying certain types of errors and modality-specific biases.
Large Multimodal Models Input Scrutiny Ability Evaluation Framework flawed premises evaluation metrics logical fallacies surface-level linguistic errors conditional flaws modality trust Gemini 2.5 pro Claude Sonnet 4 aya-vision-8b