On Large Multimodal Models as Open-World Image Classifiers
Traditional image classification requires a predefined list of semantic
categories. In contrast, Large Multimodal Models (LMMs) can sidestep this
requirement by classifying images directly using natural language (e.g.,
answering the prompt "What is the main object in the image?"). Despite this
remarkable capability, most existing studies on LMM classification performance
are surprisingly limited in scope, often assuming a closed-world setting with a
predefined set of categories. In this work, we address this gap by thoroughly
evaluating LMM classification performance in a truly open-world setting. We
first formalize the task and introduce an evaluation protocol, defining various
metrics to assess the alignment between predicted and ground truth classes. We
then evaluate 13 models across 10 benchmarks, encompassing prototypical,
non-prototypical, fine-grained, and very fine-grained classes, demonstrating
the challenges LMMs face in this task. Further analyses based on the proposed
metrics reveal the types of errors LMMs make, highlighting challenges related
to granularity and fine-grained capabilities, showing how tailored prompting
and reasoning can alleviate them.
67ea45e0cdd38e64b1134ec3
Alessandro Conti
claimed_verified
2025-03-31T08:11:29.043Z
633f243c13e836a0fc507388
https://cdn-avatars.huggingface.co/v1/production/uploads/633f243c13e836a0fc507388/Td8TWn1q2L78sZN9098AO.jpeg
Alessandro Conti
user
altndrr
67ea45e0cdd38e64b1134ec4
Massimiliano Mancini
admin_assigned
2025-03-31T10:01:57.391Z
62cf293d3200bfd438e81f1f
/avatars/608c19ee375ef091ca77d7cfbc40e76e.svg
Massimiliano Mancini
user
massimilianom
67ea45e0cdd38e64b1134ec5
Enrico Fini
67ea45e0cdd38e64b1134ec6
Yiming Wang
67ea45e0cdd38e64b1134ec7
Paolo Rota
admin_assigned
2025-03-31T10:02:22.544Z
62f3b1ea81861bd9bc5c5538
/avatars/0aef9ac5bfa91b9894166fe3c29925da.svg
Paolo Rota
user
paolorota
67ea45e0cdd38e64b1134ec8
Elisa Ricci
On Large Multimodal Models as Open-World Image Classifiers
Traditional image classification requires a predefined list of semantic
categories. In contrast, Large Multimodal Models (LMMs) can sidestep this
requirement by classifying images directly using natural language (e.g.,
answering the prompt "What is the main object in the image?"). Despite this
remarkable capability, most existing studies on LMM classification performance
are surprisingly limited in scope, often assuming a closed-world setting with a
predefined set of categories. In this work, we address this gap by thoroughly
evaluating LMM classification performance in a truly open-world setting. We
first formalize the task and introduce an evaluation protocol, defining various
metrics to assess the alignment between predicted and ground truth classes. We
then evaluate 13 models across 10 benchmarks, encompassing prototypical,
non-prototypical, fine-grained, and very fine-grained classes, demonstrating
the challenges LMMs face in this task. Further analyses based on the proposed
metrics reveal the types of errors LMMs make, highlighting challenges related
to granularity and fine-grained capabilities, showing how tailored prompting
and reasoning can alleviate them.
LMMs are evaluated for image classification in open-world settings, revealing challenges in handling granularity and fine-grained categories, and suggesting improvements through tailored prompting.
Large Multimodal Models LMMs open-world setting evaluation protocol metrics prototypical non-prototypical fine-grained very fine-grained tailored prompting reasoning