MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models
  for Integrated Capabilities
MM-Vet, with open-ended vision-language questions targeting at evaluating
integrated capabilities, has become one of the most popular benchmarks for
large multimodal model evaluation. MM-Vet assesses six core vision-language
(VL) capabilities: recognition, knowledge, spatial awareness, language
generation, OCR, and math. However, its question format is restricted to single
image-text pairs, lacking the interleaved image and text sequences prevalent in
real-world scenarios. To address this limitation, we introduce MM-Vet v2, which
includes a new VL capability called "image-text sequence understanding",
evaluating models' ability to process VL sequences. Furthermore, we maintain
the high quality of evaluation samples while further expanding the evaluation
set size. Using MM-Vet v2 to benchmark large multimodal models, we found that
Claude 3.5 Sonnet is the best model with a score of 71.8, slightly
outperforming GPT-4o which scored 71.0. Among open-weight models,
InternVL2-Llama3-76B leads with a score of 68.4.
66ac4a1b6c84fba4552abfde
Weihao Yu
admin_assigned
2024-08-02T08:24:01.048Z
5df833bdda6d0311fd3d5403
https://cdn-avatars.huggingface.co/v1/production/uploads/5df833bdda6d0311fd3d5403/62OtGJEQXdOuhV9yCd4HS.png
Weihao Yu
user
whyu
66ac4a1b6c84fba4552abfdf
Zhengyuan Yang
admin_assigned
2024-08-02T08:24:12.708Z
630713411801ecc7d2592a7c
/avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
Zhengyuan Yang
user
zyang39
66ac4a1b6c84fba4552abfe0
Linfeng Ren
66ac4a1b6c84fba4552abfe1
Linjie Li
admin_assigned
2024-08-02T08:24:42.499Z
63db16fff03c3d71ef397206
/avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
Linjie Li
user
linjieli222
66ac4a1b6c84fba4552abfe2
Jianfeng Wang
admin_assigned
2024-08-02T08:24:54.370Z
641404e74e5305c14f269b3c
/avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
Jianfeng Wang
user
amsword8
66ac4a1b6c84fba4552abfe3
Kevin Lin
admin_assigned
2024-08-02T08:25:08.200Z
6298fd95b58e71e2ac9f3ad8
/avatars/7d34644d537bc5c17cf1e4ce4095355c.svg
Kevin Lin
user
kevinlin311tw
66ac4a1b6c84fba4552abfe4
Chung-Ching Lin
claimed_verified
2025-07-23T08:39:16.057Z
6669f406d21cf7d39b1d98ba
/avatars/5ec270cb8bcb68786379d8a8bc411aaa.svg
CC Lin
user
cclin10
66ac4a1b6c84fba4552abfe5
Zicheng Liu
admin_assigned
2024-08-02T08:25:27.461Z
6569f6cb44ce94a70187f407
/avatars/414ecc9739ed23f9f395bd7cfa65055f.svg
Zicheng Liu
user
ZCLiu35
66ac4a1b6c84fba4552abfe6
Lijuan Wang
admin_assigned
2024-08-02T08:25:50.777Z
6413521d4e5305c14f22e110
/avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
Lijuan Wang
user
Lijuan
66ac4a1b6c84fba4552abfe7
Xinchao Wang
admin_assigned
2024-08-02T08:26:36.131Z
63fc03a50aab060792ffef39
/avatars/9d5b1bb2a41928e08176b703935133ab.svg
Wangxinchao
user
wxcTest
MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models
  for Integrated Capabilities
MM-Vet, with open-ended vision-language questions targeting at evaluating
integrated capabilities, has become one of the most popular benchmarks for
large multimodal model evaluation. MM-Vet assesses six core vision-language
(VL) capabilities: recognition, knowledge, spatial awareness, language
generation, OCR, and math. However, its question format is restricted to single
image-text pairs, lacking the interleaved image and text sequences prevalent in
real-world scenarios. To address this limitation, we introduce MM-Vet v2, which
includes a new VL capability called "image-text sequence understanding",
evaluating models' ability to process VL sequences. Furthermore, we maintain
the high quality of evaluation samples while further expanding the evaluation
set size. Using MM-Vet v2 to benchmark large multimodal models, we found that
Claude 3.5 Sonnet is the best model with a score of 71.8, slightly
outperforming GPT-4o which scored 71.0. Among open-weight models,
InternVL2-Llama3-76B leads with a score of 68.4.
MM-Vet v2 introduces a new benchmark for evaluating multimodal models by including image-text sequence understanding, expanding the evaluation set size, and identifying Claude 3.5 Sonnet as the top-performing model.
vision-language MM-Vet MM-Vet v2 image-text sequence understanding large multimodal models Claude 3.5 Sonnet GPT-4o InternVL2-Llama3-76B