SPARK: Multi-Vision Sensor Perception and Reasoning Benchmark for
  Large-scale Vision-Language Models
Large-scale Vision-Language Models (LVLMs) have significantly advanced with
text-aligned vision inputs. They have made remarkable progress in computer
vision tasks by aligning text modality with vision inputs. There are also
endeavors to incorporate multi-vision sensors beyond RGB, including thermal,
depth, and medical X-ray images. However, we observe that current LVLMs view
images taken from multi-vision sensors as if they were in the same RGB domain
without considering the physical characteristics of multi-vision sensors. They
fail to convey the fundamental multi-vision sensor information from the dataset
and the corresponding contextual knowledge properly. Consequently, alignment
between the information from the actual physical environment and the text is
not achieved correctly, making it difficult to answer complex sensor-related
questions that consider the physical environment. In this paper, we aim to
establish a multi-vision Sensor Perception And Reasoning benchmarK called SPARK
that can reduce the fundamental multi-vision sensor information gap between
images and multi-vision sensors. We generated 6,248 vision-language test
samples automatically to investigate multi-vision sensory perception and
multi-vision sensory reasoning on physical sensor knowledge proficiency across
different formats, covering different types of sensor-related questions. We
utilized these samples to assess ten leading LVLMs. The results showed that
most models displayed deficiencies in multi-vision sensory reasoning to varying
extents. Codes and data are available at https://github.com/top-yun/SPARK
66c7f6b9d5a538bbb9e3b387
Youngjoon Yu
66c7f6b9d5a538bbb9e3b388
Sangyun Chung
claimed_verified
2024-08-29T07:23:41.928Z
65a4bf8e90b5e87bcdff41c7
https://cdn-avatars.huggingface.co/v1/production/uploads/65a4bf8e90b5e87bcdff41c7/I6OSoZigV7Fl6OKRl_Yqy.jpeg
Sangyun Chung
user
topyun
66c7f6b9d5a538bbb9e3b389
Byung-Kwan Lee
claimed_verified
2024-08-26T07:15:42.502Z
657152eb12f162153b50ec9d
https://cdn-avatars.huggingface.co/v1/production/uploads/657152eb12f162153b50ec9d/qnldHP35PclV0pDz_05q8.jpeg
Byung-Kwan Lee
user
BK-Lee
66c7f6b9d5a538bbb9e3b38a
Yong Man Ro
SPARK: Multi-Vision Sensor Perception and Reasoning Benchmark for
  Large-scale Vision-Language Models
Large-scale Vision-Language Models (LVLMs) have significantly advanced with
text-aligned vision inputs. They have made remarkable progress in computer
vision tasks by aligning text modality with vision inputs. There are also
endeavors to incorporate multi-vision sensors beyond RGB, including thermal,
depth, and medical X-ray images. However, we observe that current LVLMs view
images taken from multi-vision sensors as if they were in the same RGB domain
without considering the physical characteristics of multi-vision sensors. They
fail to convey the fundamental multi-vision sensor information from the dataset
and the corresponding contextual knowledge properly. Consequently, alignment
between the information from the actual physical environment and the text is
not achieved correctly, making it difficult to answer complex sensor-related
questions that consider the physical environment. In this paper, we aim to
establish a multi-vision Sensor Perception And Reasoning benchmarK called SPARK
that can reduce the fundamental multi-vision sensor information gap between
images and multi-vision sensors. We generated 6,248 vision-language test
samples automatically to investigate multi-vision sensory perception and
multi-vision sensory reasoning on physical sensor knowledge proficiency across
different formats, covering different types of sensor-related questions. We
utilized these samples to assess ten leading LVLMs. The results showed that
most models displayed deficiencies in multi-vision sensory reasoning to varying
extents. Codes and data are available at https://github.com/top-yun/SPARK
A benchmark called SPARK is established to evaluate multi-vision sensory perception and reasoning in large-scale vision-language models, revealing deficiencies in understanding physical sensor characteristics.
Large-scale Vision-Language Models multi-vision sensors thermal images depth images medical X-ray images RGB domain sensor perception sensor reasoning vision-language test samples LVLMs