MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks
We present MEGA-Bench, an evaluation suite that scales multimodal evaluation
to over 500 real-world tasks, to address the highly heterogeneous daily use
cases of end users. Our objective is to optimize for a set of high-quality data
samples that cover a highly diverse and rich set of multimodal tasks, while
enabling cost-effective and accurate model evaluation. In particular, we
collected 505 realistic tasks encompassing over 8,000 samples from 16 expert
annotators to extensively cover the multimodal task space. Instead of unifying
these problems into standard multi-choice questions (like MMMU, MMBench, and
MMT-Bench), we embrace a wide range of output formats like numbers, phrases,
code, \LaTeX, coordinates, JSON, free-form, etc. To accommodate these formats,
we developed over 40 metrics to evaluate these tasks. Unlike existing
benchmarks, MEGA-Bench offers a fine-grained capability report across multiple
dimensions (e.g., application, input type, output format, skill), allowing
users to interact with and visualize model capabilities in depth. We evaluate a
wide variety of frontier vision-language models on MEGA-Bench to understand
their capabilities across these dimensions.
670dd82d85a1548f5cdf99fd
Jiacheng Chen
admin_assigned
2024-10-15T13:32:05.739Z
655bca95360e4f90cb61ba83
/avatars/1a187beb91a5e2fdc2303620b742aab1.svg
Jiacheng Chen
user
cccjc
670dd82d85a1548f5cdf99fe
Tianhao Liang
claimed_verified
2024-10-21T07:48:45.081Z
6703ec213df5fe425086ef73
/avatars/e6f9dad6587ee0883ae10f8805ab7ea9.svg
Tianhao Liang
user
tianhao2k
670dd82d85a1548f5cdf99ff
Sherman Siu
claimed_verified
2024-10-15T09:10:52.899Z
630d73387dacb93b335b00e9
/avatars/2ded3550aedf77a58175a66319d01820.svg
Sherman Siu
user
shermansiu
670dd82d85a1548f5cdf9a00
Zhengqing Wang
admin_assigned
2024-10-15T13:33:34.951Z
64d97c5bfd0b55d501ba00cf
/avatars/47505f2a573acea7176a96f538226ecb.svg
Zhengqing Wang
user
EricW123456
670dd82d85a1548f5cdf9a01
Kai Wang
670dd82d85a1548f5cdf9a02
Yubo Wang
claimed_verified
2025-01-30T08:41:25.642Z
636a35eff8d9af4aea181608
https://cdn-avatars.huggingface.co/v1/production/uploads/636a35eff8d9af4aea181608/s9GFJYd_QXVbg0Lb4JpKj.jpeg
yubo
user
ubowang
670dd82d85a1548f5cdf9a03
Yuansheng Ni
claimed_verified
2024-10-15T09:10:49.671Z
64de37ee5e192985054be575
https://cdn-avatars.huggingface.co/v1/production/uploads/64de37ee5e192985054be575/fVV7JQMtp_J3uFqszJJHH.jpeg
Yuansheng Ni
user
yuanshengni
670dd82d85a1548f5cdf9a04
Wang Zhu
claimed_verified
2024-11-26T21:21:31.293Z
6389568eec1f539adc0e0337
/avatars/6ee5c9bea9989e4618ae8d334bd889a9.svg
Wang Bill Zhu
user
Bill1235813
670dd82d85a1548f5cdf9a05
Ziyan Jiang
claimed_verified
2024-10-15T13:06:01.261Z
64778fb8168cb428e00f69b0
https://cdn-avatars.huggingface.co/v1/production/uploads/64778fb8168cb428e00f69b0/D_XYg74zHG9K3HUJj_gD4.jpeg
Ziyan Jiang
user
ziyjiang
670dd82d85a1548f5cdf9a06
Bohan Lyu
claimed_verified
2024-11-03T14:24:09.935Z
650267e7e751d03da933a24a
/avatars/f047a047d1de304cd97027463541bdf3.svg
Bohan22
user
Bohan22
670dd82d85a1548f5cdf9a07
Dongfu Jiang
claimed_verified
2024-10-15T09:10:51.346Z
62567c86d444a9b5a0ec51c1
https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
Dongfu Jiang
user
DongfuJiang
670dd82d85a1548f5cdf9a08
Xuan He
670dd82d85a1548f5cdf9a09
Yuan Liu
admin_assigned
2024-10-15T13:31:06.125Z
64d47a7a508a6313e33faedd
https://cdn-avatars.huggingface.co/v1/production/uploads/64d47a7a508a6313e33faedd/8DrbXdxA-sKZb4jgNhvY1.jpeg
Yuan Liu
user
YuanLiuuuuuu
670dd82d85a1548f5cdf9a0a
Hexiang Hu
admin_assigned
2024-10-15T13:30:44.118Z
643441ccd55dea2d0ec2c309
/avatars/82e99d445e2b513ad7270fa852adbcbb.svg
Hexiang Hu
user
hexianghu
670dd82d85a1548f5cdf9a0b
Xiang Yue
claimed_verified
2024-10-15T09:10:54.823Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
670dd82d85a1548f5cdf9a0c
Wenhu Chen
extracted_pending
2024-10-15T02:49:20.901Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks
We present MEGA-Bench, an evaluation suite that scales multimodal evaluation
to over 500 real-world tasks, to address the highly heterogeneous daily use
cases of end users. Our objective is to optimize for a set of high-quality data
samples that cover a highly diverse and rich set of multimodal tasks, while
enabling cost-effective and accurate model evaluation. In particular, we
collected 505 realistic tasks encompassing over 8,000 samples from 16 expert
annotators to extensively cover the multimodal task space. Instead of unifying
these problems into standard multi-choice questions (like MMMU, MMBench, and
MMT-Bench), we embrace a wide range of output formats like numbers, phrases,
code, \LaTeX, coordinates, JSON, free-form, etc. To accommodate these formats,
we developed over 40 metrics to evaluate these tasks. Unlike existing
benchmarks, MEGA-Bench offers a fine-grained capability report across multiple
dimensions (e.g., application, input type, output format, skill), allowing
users to interact with and visualize model capabilities in depth. We evaluate a
wide variety of frontier vision-language models on MEGA-Bench to understand
their capabilities across these dimensions.
MEGA-Bench evaluates vision-language models across over 500 real-world tasks with diverse output formats, providing a comprehensive capability report.
multimodal evaluation real-world tasks expert annotators output formats metrics capability report vision-language models