MMMG: a Comprehensive and Reliable Evaluation Suite for Multitask
  Multimodal Generation
Automatically evaluating multimodal generation presents a significant
challenge, as automated metrics often struggle to align reliably with human
evaluation, especially for complex tasks that involve multiple modalities. To
address this, we present MMMG, a comprehensive and human-aligned benchmark for
multimodal generation across 4 modality combinations (image, audio, interleaved
text and image, interleaved text and audio), with a focus on tasks that present
significant challenges for generation models, while still enabling reliable
automatic evaluation through a combination of models and programs. MMMG
encompasses 49 tasks (including 29 newly developed ones), each with a carefully
designed evaluation pipeline, and 937 instructions to systematically assess
reasoning, controllability, and other key capabilities of multimodal generation
models. Extensive validation demonstrates that MMMG is highly aligned with
human evaluation, achieving an average agreement of 94.3%. Benchmarking results
on 24 multimodal generation models reveal that even though the state-of-the-art
model, GPT Image, achieves 78.3% accuracy for image generation, it falls short
on multimodal reasoning and interleaved generation. Furthermore, results
suggest considerable headroom for improvement in audio generation, highlighting
an important direction for future research.
68361038450a48f228ff555e
Jihan Yao
claimed_verified
2025-05-28T19:14:25.358Z
65750954a90ae2daaec7ab93
/avatars/43ea342d9a10caeadc0183afc3848f5a.svg
Jihan Yao
user
yaojh18
68361038450a48f228ff555f
Yushi Hu
extracted_confirmed
2025-05-27T19:21:19.936Z
62b1474bdcbad6848a91a54e
/avatars/d7308899b46232cad4a48a0e876449a8.svg
Yushi Hu
user
yushihu
68361038450a48f228ff5560
Yujie Yi
68361038450a48f228ff5561
Bin Han
68361038450a48f228ff5562
Shangbin Feng
68361038450a48f228ff5563
Guang Yang
68361038450a48f228ff5564
Bingbing Wen
68361038450a48f228ff5565
Ranjay Krishna
68361038450a48f228ff5566
Lucy Lu Wang
68361038450a48f228ff5567
Yulia Tsvetkov
68361038450a48f228ff5568
Noah A. Smith
68361038450a48f228ff5569
Banghua Zhu
MMMG: a Comprehensive and Reliable Evaluation Suite for Multitask
  Multimodal Generation
Automatically evaluating multimodal generation presents a significant
challenge, as automated metrics often struggle to align reliably with human
evaluation, especially for complex tasks that involve multiple modalities. To
address this, we present MMMG, a comprehensive and human-aligned benchmark for
multimodal generation across 4 modality combinations (image, audio, interleaved
text and image, interleaved text and audio), with a focus on tasks that present
significant challenges for generation models, while still enabling reliable
automatic evaluation through a combination of models and programs. MMMG
encompasses 49 tasks (including 29 newly developed ones), each with a carefully
designed evaluation pipeline, and 937 instructions to systematically assess
reasoning, controllability, and other key capabilities of multimodal generation
models. Extensive validation demonstrates that MMMG is highly aligned with
human evaluation, achieving an average agreement of 94.3%. Benchmarking results
on 24 multimodal generation models reveal that even though the state-of-the-art
model, GPT Image, achieves 78.3% accuracy for image generation, it falls short
on multimodal reasoning and interleaved generation. Furthermore, results
suggest considerable headroom for improvement in audio generation, highlighting
an important direction for future research.
MMMG is a comprehensive benchmark for multimodal generation, offering 49 tasks and 937 instructions to align automatic evaluation with human judgment, revealing areas for improvement in reasoning and audio generation.
multimodal generation evaluation pipeline human-aligned benchmark multimodal reasoning interleaved generation audio generation GPT Image