On the Limitations of Vision-Language Models in Understanding Image
  Transforms
Vision Language Models (VLMs) have demonstrated significant potential in
various downstream tasks, including Image/Video Generation, Visual Question
Answering, Multimodal Chatbots, and Video Understanding. However, these models
often struggle with basic image transformations. This paper investigates the
image-level understanding of VLMs, specifically CLIP by OpenAI and SigLIP by
Google. Our findings reveal that these models lack comprehension of multiple
image-level augmentations. To facilitate this study, we created an augmented
version of the Flickr8k dataset, pairing each image with a detailed description
of the applied transformation. We further explore how this deficiency impacts
downstream tasks, particularly in image editing, and evaluate the performance
of state-of-the-art Image2Image models on simple transformations.
67d4194fc9f149faff46913c
Ahmad Mustafa Anis
claimed_verified
2025-03-16T21:14:01.381Z
6246908d8031dcfa9ef6d80b
https://cdn-avatars.huggingface.co/v1/production/uploads/6246908d8031dcfa9ef6d80b/hVdURjUl1RS2MZf4qOhvI.jpeg
Ahmad Mustafa Anis
user
AhmadMustafa
67d4194fc9f149faff46913d
Hasnain Ali
67d4194fc9f149faff46913e
Saquib Sarfraz
On the Limitations of Vision-Language Models in Understanding Image
  Transforms
Vision Language Models (VLMs) have demonstrated significant potential in
various downstream tasks, including Image/Video Generation, Visual Question
Answering, Multimodal Chatbots, and Video Understanding. However, these models
often struggle with basic image transformations. This paper investigates the
image-level understanding of VLMs, specifically CLIP by OpenAI and SigLIP by
Google. Our findings reveal that these models lack comprehension of multiple
image-level augmentations. To facilitate this study, we created an augmented
version of the Flickr8k dataset, pairing each image with a detailed description
of the applied transformation. We further explore how this deficiency impacts
downstream tasks, particularly in image editing, and evaluate the performance
of state-of-the-art Image2Image models on simple transformations.
Vision Language Models (VLMs) Image/Video Generation Visual Question Answering Multimodal Chatbots Video Understanding Image-level Understanding CLIP SigLIP Image-level augmentations Flickr8k dataset Image2Image models