UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond
  Scaling
Significant research efforts have been made to scale and improve
vision-language model (VLM) training approaches. Yet, with an ever-growing
number of benchmarks, researchers are tasked with the heavy burden of
implementing each protocol, bearing a non-trivial computational cost, and
making sense of how all these benchmarks translate into meaningful axes of
progress. To facilitate a systematic evaluation of VLM progress, we introduce
UniBench: a unified implementation of 50+ VLM benchmarks spanning a
comprehensive range of carefully categorized capabilities from object
recognition to spatial awareness, counting, and much more. We showcase the
utility of UniBench for measuring progress by evaluating nearly 60 publicly
available vision-language models, trained on scales of up to 12.8B samples. We
find that while scaling training data or model size can boost many
vision-language model capabilities, scaling offers little benefit for reasoning
or relations. Surprisingly, we also discover today's best VLMs struggle on
simple digit recognition and counting tasks, e.g. MNIST, which much simpler
networks can solve. Where scale falls short, we find that more precise
interventions, such as data quality or tailored-learning objectives offer more
promise. For practitioners, we also offer guidance on selecting a suitable VLM
for a given application. Finally, we release an easy-to-run UniBench code-base
with the full set of 50+ benchmarks and comparisons across 59 models as well as
a distilled, representative set of benchmarks that runs in 5 minutes on a
single GPU.
66b97d8af1dfeeafae99200f
Haider Al-Tahan
extracted_pending
2024-08-12T03:12:13.598Z
63654c5edaa65e06f90cc5ee
/avatars/11fd0c361531f97b16b91629d0083b92.svg
haider
user
haideraltahan
66b97d8af1dfeeafae992010
Quentin Garrido
admin_assigned
2024-08-13T11:11:12.158Z
63049022412a1b9d381b9dcb
/avatars/cac2a540fda40c5ae558968c50385f46.svg
Quentin Garrido
user
garridoq
66b97d8af1dfeeafae992011
Randall Balestriero
admin_assigned
2024-08-13T11:11:18.506Z
6629cf0121e410e67c4de179
/avatars/157637f7716849675fe44da005fd3ede.svg
Randall Balestriero
user
RandallBalestriero
66b97d8af1dfeeafae992012
Diane Bouchacourt
admin_assigned
2024-08-13T11:11:24.143Z
641e01d13bae5a776368311b
/avatars/1e983fb59f475b3252e201f44eec4657.svg
Diane Bouchacourt
user
DianeB
66b97d8af1dfeeafae992013
Caner Hazirbas
admin_assigned
2024-08-13T11:11:29.894Z
6515b9414eb20107d01eba58
https://cdn-avatars.huggingface.co/v1/production/uploads/6515b9414eb20107d01eba58/8LSqNIgItKUVkWlEQERMD.jpeg
Caner Hazirbas
user
hazirbas
66b97d8af1dfeeafae992014
Mark Ibrahim
claimed_verified
2024-11-18T15:20:04.874Z
64c17345e82e55936cf971bc
/avatars/b39c5637e00291410620afb7c770d587.svg
Mark Ibrahim
user
marksibrahim
UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond
  Scaling
Significant research efforts have been made to scale and improve
vision-language model (VLM) training approaches. Yet, with an ever-growing
number of benchmarks, researchers are tasked with the heavy burden of
implementing each protocol, bearing a non-trivial computational cost, and
making sense of how all these benchmarks translate into meaningful axes of
progress. To facilitate a systematic evaluation of VLM progress, we introduce
UniBench: a unified implementation of 50+ VLM benchmarks spanning a
comprehensive range of carefully categorized capabilities from object
recognition to spatial awareness, counting, and much more. We showcase the
utility of UniBench for measuring progress by evaluating nearly 60 publicly
available vision-language models, trained on scales of up to 12.8B samples. We
find that while scaling training data or model size can boost many
vision-language model capabilities, scaling offers little benefit for reasoning
or relations. Surprisingly, we also discover today's best VLMs struggle on
simple digit recognition and counting tasks, e.g. MNIST, which much simpler
networks can solve. Where scale falls short, we find that more precise
interventions, such as data quality or tailored-learning objectives offer more
promise. For practitioners, we also offer guidance on selecting a suitable VLM
for a given application. Finally, we release an easy-to-run UniBench code-base
with the full set of 50+ benchmarks and comparisons across 59 models as well as
a distilled, representative set of benchmarks that runs in 5 minutes on a
single GPU.
UniBench provides a unified evaluation framework for vision-language models across various benchmarks, revealing that scaling does not always improve reasoning capabilities and simpler networks can outperform large models on basic tasks.
vision-language model (VLM) UniBench benchmarks object recognition spatial awareness counting data quality learning objectives