LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
The advances of large foundation models necessitate wide-coverage, low-cost,
and zero-contamination benchmarks. Despite continuous exploration of language
model evaluations, comprehensive studies on the evaluation of Large Multi-modal
Models (LMMs) remain limited. In this work, we introduce LMMS-EVAL, a unified
and standardized multimodal benchmark framework with over 50 tasks and more
than 10 models to promote transparent and reproducible evaluations. Although
LMMS-EVAL offers comprehensive coverage, we find it still falls short in
achieving low cost and zero contamination. To approach this evaluation
trilemma, we further introduce LMMS-EVAL LITE, a pruned evaluation toolkit that
emphasizes both coverage and efficiency. Additionally, we present Multimodal
LIVEBENCH that utilizes continuously updating news and online forums to assess
models' generalization abilities in the wild, featuring a low-cost and
zero-contamination evaluation approach. In summary, our work highlights the
importance of considering the evaluation trilemma and provides practical
solutions to navigate the trade-offs in evaluating large multi-modal models,
paving the way for more effective and reliable benchmarking of LMMs. We
opensource our codebase and maintain leaderboard of LIVEBENCH at
https://github.com/EvolvingLMMs-Lab/lmms-eval and
https://huggingface.co/spaces/lmms-lab/LiveBench.
66987d94a029d7f9e39da94f
Kaichen Zhang
claimed_verified
2024-07-18T09:05:28.420Z
64bb77e786e7fb5b8a317a43
https://cdn-avatars.huggingface.co/v1/production/uploads/64bb77e786e7fb5b8a317a43/J0jOrlZJ9gazdYaeSH2Bo.png
kcz
user
kcz358
66987d94a029d7f9e39da950
Bo Li
claimed_verified
2024-08-09T07:48:40.807Z
62d3f7d84b0933c48f3cdd9c
https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
Bo Li
user
luodian
66987d94a029d7f9e39da951
Peiyuan Zhang
claimed_verified
2024-07-18T09:05:25.903Z
63565cc56d7fcf1bedb7d347
https://cdn-avatars.huggingface.co/v1/production/uploads/63565cc56d7fcf1bedb7d347/XGcHP4VkO_oieA1gZ4IAX.jpeg
Zhang Peiyuan
user
PY007
66987d94a029d7f9e39da952
Fanyi Pu
claimed_verified
2024-07-22T07:10:47.248Z
646e1ef5075bbcc48ddf21e8
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_vJC0zeVOIvaNV2R6toqg.jpeg
Pu Fanyi
user
pufanyi
66987d94a029d7f9e39da953
Joshua Adrian Cahyono
66987d94a029d7f9e39da954
Kairui Hu
claimed_verified
2025-01-24T09:08:32.574Z
6400ba2b261cfa61f3a00555
/avatars/1311e0b5e21b1c94d73fcaf455d3c7f7.svg
Kairui
user
KairuiHu
66987d94a029d7f9e39da955
Shuai Liu
claimed_verified
2025-03-07T13:38:17.038Z
64f7f5b54101c731ca84ae05
https://cdn-avatars.huggingface.co/v1/production/uploads/64f7f5b54101c731ca84ae05/13DwdxOo3tWbxKDLd44B9.jpeg
Shuai Liu
user
Choiszt
66987d94a029d7f9e39da956
Yuanhan Zhang
claimed_verified
2024-07-19T11:23:46.860Z
62a993d80472c0b7f94027df
https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
Zhang Yuanhan
user
ZhangYuanhan
66987d94a029d7f9e39da957
Jingkang Yang
66987d94a029d7f9e39da958
Chunyuan Li
66987d94a029d7f9e39da959
Ziwei Liu
LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
The advances of large foundation models necessitate wide-coverage, low-cost,
and zero-contamination benchmarks. Despite continuous exploration of language
model evaluations, comprehensive studies on the evaluation of Large Multi-modal
Models (LMMs) remain limited. In this work, we introduce LMMS-EVAL, a unified
and standardized multimodal benchmark framework with over 50 tasks and more
than 10 models to promote transparent and reproducible evaluations. Although
LMMS-EVAL offers comprehensive coverage, we find it still falls short in
achieving low cost and zero contamination. To approach this evaluation
trilemma, we further introduce LMMS-EVAL LITE, a pruned evaluation toolkit that
emphasizes both coverage and efficiency. Additionally, we present Multimodal
LIVEBENCH that utilizes continuously updating news and online forums to assess
models' generalization abilities in the wild, featuring a low-cost and
zero-contamination evaluation approach. In summary, our work highlights the
importance of considering the evaluation trilemma and provides practical
solutions to navigate the trade-offs in evaluating large multi-modal models,
paving the way for more effective and reliable benchmarking of LMMs. We
opensource our codebase and maintain leaderboard of LIVEBENCH at
https://github.com/EvolvingLMMs-Lab/lmms-eval and
https://huggingface.co/spaces/lmms-lab/LiveBench.
LMMS-EVAL and LMMS-EVAL LITE provide frameworks for evaluating large multi-modal models with comprehensive coverage, while Multimodal LIVEBENCH assesses models' generalization using real-world data.
Large Multi-modal Models LMMS-EVAL LMMS-EVAL LITE Multimodal LIVEBENCH multimodal benchmark