MME-Unify: A Comprehensive Benchmark for Unified Multimodal
  Understanding and Generation Models
Existing MLLM benchmarks face significant challenges in evaluating Unified
MLLMs (U-MLLMs) due to: 1) lack of standardized benchmarks for traditional
tasks, leading to inconsistent comparisons; 2) absence of benchmarks for
mixed-modality generation, which fails to assess multimodal reasoning
capabilities. We present a comprehensive evaluation framework designed to
systematically assess U-MLLMs. Our benchmark includes: Standardized Traditional
Task Evaluation. We sample from 12 datasets, covering 10 tasks with 30
subtasks, ensuring consistent and fair comparisons across studies." 2. Unified
Task Assessment. We introduce five novel tasks testing multimodal reasoning,
including image editing, commonsense QA with image generation, and geometric
reasoning. 3. Comprehensive Model Benchmarking. We evaluate 12 leading U-MLLMs,
such as Janus-Pro, EMU3, VILA-U, and Gemini2-flash, alongside specialized
understanding (e.g., Claude-3.5-Sonnet) and generation models (e.g., DALL-E-3).
Our findings reveal substantial performance gaps in existing U-MLLMs,
highlighting the need for more robust models capable of handling mixed-modality
tasks effectively. The code and evaluation data can be found in
https://mme-unify.github.io/.
67f34f5dfb6d8a613926ac2b
Wulin Xie
67f34f5dfb6d8a613926ac2c
Yi-Fan Zhang
claimed_verified
2025-04-09T07:38:04.393Z
623d8ca4c29adf5ef6175615
https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
Yi-Fan Zhang
user
yifanzhang114
67f34f5dfb6d8a613926ac2d
Chaoyou Fu
67f34f5dfb6d8a613926ac2e
Yang Shi
claimed_verified
2025-04-08T06:53:09.884Z
673c7319d11b1c2e246ead9c
https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg
Yang Shi
user
DogNeverSleep
67f34f5dfb6d8a613926ac2f
Bingyan Nie
67f34f5dfb6d8a613926ac30
Hongkai Chen
67f34f5dfb6d8a613926ac31
Zhang Zhang
67f34f5dfb6d8a613926ac32
Liang Wang
67f34f5dfb6d8a613926ac33
Tieniu Tan
MME-Unify: A Comprehensive Benchmark for Unified Multimodal
  Understanding and Generation Models
Existing MLLM benchmarks face significant challenges in evaluating Unified
MLLMs (U-MLLMs) due to: 1) lack of standardized benchmarks for traditional
tasks, leading to inconsistent comparisons; 2) absence of benchmarks for
mixed-modality generation, which fails to assess multimodal reasoning
capabilities. We present a comprehensive evaluation framework designed to
systematically assess U-MLLMs. Our benchmark includes: Standardized Traditional
Task Evaluation. We sample from 12 datasets, covering 10 tasks with 30
subtasks, ensuring consistent and fair comparisons across studies." 2. Unified
Task Assessment. We introduce five novel tasks testing multimodal reasoning,
including image editing, commonsense QA with image generation, and geometric
reasoning. 3. Comprehensive Model Benchmarking. We evaluate 12 leading U-MLLMs,
such as Janus-Pro, EMU3, VILA-U, and Gemini2-flash, alongside specialized
understanding (e.g., Claude-3.5-Sonnet) and generation models (e.g., DALL-E-3).
Our findings reveal substantial performance gaps in existing U-MLLMs,
highlighting the need for more robust models capable of handling mixed-modality
tasks effectively. The code and evaluation data can be found in
https://mme-unify.github.io/.
A new evaluation framework assesses Unified MLLMs across traditional and mixed-modality tasks, revealing performance gaps and the need for more robust models.
Unified MLLMs U-MLLMs standardized benchmarks mixed-modality generation multimodal reasoning image editing commonsense QA geometric reasoning Janus-Pro EMU3 VILA-U Gemini2-flash Claude-3.5-Sonnet DALL-E-3