GIR-Bench: Versatile Benchmark for Generating Images with Reasoning
Unified multimodal models integrate the reasoning capacity of large language
models with both image understanding and generation, showing great promise for
advanced multimodal intelligence. However, the community still lacks a rigorous
reasoning-centric benchmark to systematically evaluate the alignment between
understanding and generation, and their generalization potential in complex
visual tasks. To this end, we introduce GIR-Bench, a comprehensive
benchmark that evaluates unified models across three complementary
perspectives. Firstly, we investigate understanding-generation consistency
(GIR-Bench-UGC), asking whether models can consistently leverage the same
knowledge in both understanding and generation tasks. Secondly, we investigate
whether models can perform reasoning-centric text-to-image generation that
requires applying logical constraints and implicit knowledge to generate
faithful visual content (GIR-Bench-T2I). Thirdly, we evaluate whether models
can handle multi-step reasoning in editing (GIR-Bench-Edit). For each subset,
we carefully design different task-specific evaluation pipelines tailored for
each task. This enables fine-grained and interpretable evaluation while
mitigating biases from the prevalent MLLM-as-a-Judge paradigm. Extensive
ablations over various unified models and generation-only systems have shown
that: Although unified models are more capable of reasoning-driven visual
tasks, they still exhibit a persistent gap between understanding and
generation. The data and code for GIR-Bench are available at
https://hkust-longgroup.github.io/GIR-Bench{https://hkust-longgroup.github.io/GIR-Bench}.
68edc362de1fee572713a8bb
Hongxiang Li
claimed_verified
2025-10-14T07:30:17.520Z
675b94e3a301c0203a4131bc
/avatars/4b7b74a3efb1ed1649114c644389b8c9.svg
Hongxiang Li
user
lihxxx
68edc362de1fee572713a8bc
Yaowei Li
68edc362de1fee572713a8bd
Bin Lin
claimed_verified
2025-10-21T14:06:31.411Z
6367a8175bb06007ea099b8f
https://cdn-avatars.huggingface.co/v1/production/uploads/6367a8175bb06007ea099b8f/IjG7HyWyWRlVt_XwRbxRW.jpeg
linbin
user
LanguageBind
68edc362de1fee572713a8be
Yuwei Niu
68edc362de1fee572713a8bf
Yuhang Yang
68edc362de1fee572713a8c0
Xiaoshuang Huang
68edc362de1fee572713a8c1
Jiayin Cai
68edc362de1fee572713a8c2
Xiaolong Jiang
68edc362de1fee572713a8c3
Yao Hu
68edc362de1fee572713a8c4
Long Chen
GIR-Bench: Versatile Benchmark for Generating Images with Reasoning
Unified multimodal models integrate the reasoning capacity of large language
models with both image understanding and generation, showing great promise for
advanced multimodal intelligence. However, the community still lacks a rigorous
reasoning-centric benchmark to systematically evaluate the alignment between
understanding and generation, and their generalization potential in complex
visual tasks. To this end, we introduce GIR-Bench, a comprehensive
benchmark that evaluates unified models across three complementary
perspectives. Firstly, we investigate understanding-generation consistency
(GIR-Bench-UGC), asking whether models can consistently leverage the same
knowledge in both understanding and generation tasks. Secondly, we investigate
whether models can perform reasoning-centric text-to-image generation that
requires applying logical constraints and implicit knowledge to generate
faithful visual content (GIR-Bench-T2I). Thirdly, we evaluate whether models
can handle multi-step reasoning in editing (GIR-Bench-Edit). For each subset,
we carefully design different task-specific evaluation pipelines tailored for
each task. This enables fine-grained and interpretable evaluation while
mitigating biases from the prevalent MLLM-as-a-Judge paradigm. Extensive
ablations over various unified models and generation-only systems have shown
that: Although unified models are more capable of reasoning-driven visual
tasks, they still exhibit a persistent gap between understanding and
generation. The data and code for GIR-Bench are available at
https://hkust-longgroup.github.io/GIR-Bench{https://hkust-longgroup.github.io/GIR-Bench}.
GIR-Bench evaluates unified multimodal models across understanding-generation consistency, reasoning-centric text-to-image generation, and multi-step reasoning in editing, highlighting gaps in their capabilities.
unified multimodal models large language models image understanding image generation multimodal intelligence reasoning-centric benchmark understanding-generation consistency text-to-image generation multi-step reasoning evaluation pipelines MLLM-as-a-Judge paradigm