How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation
  Models on Standard Computer Vision Tasks
Multimodal foundation models, such as GPT-4o, have recently made remarkable
progress, but it is not clear where exactly these models stand in terms of
understanding vision. In this paper, we benchmark the performance of popular
multimodal foundation models (GPT-4o, o4-mini, Gemini 1.5 Pro and Gemini 2.0
Flash, Claude 3.5 Sonnet, Qwen2-VL, Llama 3.2) on standard computer vision
tasks (semantic segmentation, object detection, image classification, depth and
surface normal prediction) using established datasets (e.g., COCO, ImageNet and
its variants, etc).
  The main challenges to performing this are: 1) most models are trained to
output text and cannot natively express versatile domains, such as segments or
3D geometry, and 2) many leading models are proprietary and accessible only at
an API level, i.e., there is no weight access to adapt them. We address these
challenges by translating standard vision tasks into equivalent text-promptable
and API-compatible tasks via prompt chaining to create a standardized
benchmarking framework.
  We observe that 1) the models are not close to the state-of-the-art
specialist models at any task. However, 2) they are respectable generalists;
this is remarkable as they are presumably trained on primarily image-text-based
tasks. 3) They perform semantic tasks notably better than geometric ones. 4)
While the prompt-chaining techniques affect performance, better models exhibit
less sensitivity to prompt variations. 5) GPT-4o performs the best among
non-reasoning models, securing the top position in 4 out of 6 tasks, 6)
reasoning models, e.g. o3, show improvements in geometric tasks, and 7) a
preliminary analysis of models with native image generation, like the latest
GPT-4o, shows they exhibit quirks like hallucinations and spatial
misalignments.
686b8347213f123a1f88bdc8
Rahul Ramachandran
686b8347213f123a1f88bdc9
Ali Garjani
686b8347213f123a1f88bdca
Roman Bachmann
686b8347213f123a1f88bdcb
Andrei Atanov
686b8347213f123a1f88bdcc
Oğuzhan Fatih Kar
686b8347213f123a1f88bdcd
Amir Zamir
How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation
  Models on Standard Computer Vision Tasks
Multimodal foundation models, such as GPT-4o, have recently made remarkable
progress, but it is not clear where exactly these models stand in terms of
understanding vision. In this paper, we benchmark the performance of popular
multimodal foundation models (GPT-4o, o4-mini, Gemini 1.5 Pro and Gemini 2.0
Flash, Claude 3.5 Sonnet, Qwen2-VL, Llama 3.2) on standard computer vision
tasks (semantic segmentation, object detection, image classification, depth and
surface normal prediction) using established datasets (e.g., COCO, ImageNet and
its variants, etc).
  The main challenges to performing this are: 1) most models are trained to
output text and cannot natively express versatile domains, such as segments or
3D geometry, and 2) many leading models are proprietary and accessible only at
an API level, i.e., there is no weight access to adapt them. We address these
challenges by translating standard vision tasks into equivalent text-promptable
and API-compatible tasks via prompt chaining to create a standardized
benchmarking framework.
  We observe that 1) the models are not close to the state-of-the-art
specialist models at any task. However, 2) they are respectable generalists;
this is remarkable as they are presumably trained on primarily image-text-based
tasks. 3) They perform semantic tasks notably better than geometric ones. 4)
While the prompt-chaining techniques affect performance, better models exhibit
less sensitivity to prompt variations. 5) GPT-4o performs the best among
non-reasoning models, securing the top position in 4 out of 6 tasks, 6)
reasoning models, e.g. o3, show improvements in geometric tasks, and 7) a
preliminary analysis of models with native image generation, like the latest
GPT-4o, shows they exhibit quirks like hallucinations and spatial
misalignments.
Popular multimodal foundation models perform respectably across various computer vision tasks when adapted through prompt chaining, though they fall short of specialist models and exhibit quirks in image generation.
multimodal foundation models GPT-4o o4-mini Gemini 1.5 Pro Gemini 2.0 Flash Claude 3.5 Sonnet Qwen2-VL Llama 3.2 semantic segmentation object detection image classification depth prediction surface normal prediction COCO ImageNet prompt chaining reasoning models hallucinations spatial misalignments