GRAB: A Challenging GRaph Analysis Benchmark for Large Multimodal Models
Large multimodal models (LMMs) have exhibited proficiencies across many
visual tasks. Although numerous well-known benchmarks exist to evaluate model
performance, they increasingly have insufficient headroom. As such, there is a
pressing need for a new generation of benchmarks challenging enough for the
next generation of LMMs. One area that LMMs show potential is graph analysis,
specifically, the tasks an analyst might typically perform when interpreting
figures such as estimating the mean, intercepts or correlations of functions
and data series. In this work, we introduce GRAB, a graph analysis benchmark,
fit for current and future frontier LMMs. Our benchmark is entirely synthetic,
ensuring high-quality, noise-free questions. GRAB is comprised of 2170
questions, covering four tasks and 23 graph properties. We evaluate 20 LMMs on
GRAB, finding it to be a challenging benchmark, with the highest performing
model attaining a score of just 21.7%. Finally, we conduct various ablations to
investigate where the models succeed and struggle. We release GRAB to encourage
progress in this important, growing domain.
66c6eb81e41514386f743ce6
Jonathan Roberts
claimed_verified
2024-08-22T19:50:17.177Z
632456d21ed511c0c5231afd
https://cdn-avatars.huggingface.co/v1/production/uploads/1671535517748-632456d21ed511c0c5231afd.jpeg
Jonathan Roberts
user
jonathan-roberts1
66c6eb81e41514386f743ce7
Kai Han
66c6eb81e41514386f743ce8
Samuel Albanie
GRAB: A Challenging GRaph Analysis Benchmark for Large Multimodal Models
Large multimodal models (LMMs) have exhibited proficiencies across many
visual tasks. Although numerous well-known benchmarks exist to evaluate model
performance, they increasingly have insufficient headroom. As such, there is a
pressing need for a new generation of benchmarks challenging enough for the
next generation of LMMs. One area that LMMs show potential is graph analysis,
specifically, the tasks an analyst might typically perform when interpreting
figures such as estimating the mean, intercepts or correlations of functions
and data series. In this work, we introduce GRAB, a graph analysis benchmark,
fit for current and future frontier LMMs. Our benchmark is entirely synthetic,
ensuring high-quality, noise-free questions. GRAB is comprised of 2170
questions, covering four tasks and 23 graph properties. We evaluate 20 LMMs on
GRAB, finding it to be a challenging benchmark, with the highest performing
model attaining a score of just 21.7%. Finally, we conduct various ablations to
investigate where the models succeed and struggle. We release GRAB to encourage
progress in this important, growing domain.
GRAB is a synthetic graph analysis benchmark designed to challenge large multimodal models (LMMs) across various tasks and graph properties, with the top-performing model achieving only 21.7% accuracy.
large multimodal models graph analysis synthetic benchmark graph properties ablations