RBench-V: A Primary Assessment for Visual Reasoning Models with
  Multi-modal Outputs
The rapid advancement of native multi-modal models and omni-models,
exemplified by GPT-4o, Gemini, and o3, with their capability to process and
generate content across modalities such as text and images, marks a significant
milestone in the evolution of intelligence. Systematic evaluation of their
multi-modal output capabilities in visual thinking processes (also known as
multi-modal chain of thought, M-CoT) becomes critically important. However,
existing benchmarks for evaluating multi-modal models primarily focus on
assessing multi-modal inputs and text-only reasoning while neglecting the
importance of reasoning through multi-modal outputs. In this paper, we present
a benchmark, dubbed RBench-V, designed to assess models' vision-indispensable
reasoning abilities. To construct RBench-V, we carefully hand-pick 803
questions covering math, physics, counting, and games. Unlike previous
benchmarks that typically specify certain input modalities, RBench-V presents
problems centered on multi-modal outputs, which require image manipulation such
as generating novel images and constructing auxiliary lines to support the
reasoning process. We evaluate numerous open- and closed-source models on
RBench-V, including o3, Gemini 2.5 Pro, Qwen2.5-VL, etc. Even the
best-performing model, o3, achieves only 25.8% accuracy on RBench-V, far below
the human score of 82.3%, highlighting that current models struggle to leverage
multi-modal reasoning. Data and code are available at
https://evalmodels.github.io/rbenchv
68308e2697d9a81c8521bc6a
Meng-Hao Guo
claimed_verified
2025-05-26T08:12:57.462Z
62145614b670cb63a38075ba
/avatars/5e33debde75ae6c87640f63c48c560c6.svg
MenghaoGuo
user
MenghaoGuo
68308e2697d9a81c8521bc6b
Xuanyu Chu
claimed_verified
2025-05-26T08:12:48.661Z
66b711f9512dac2ac08bc5e5
https://cdn-avatars.huggingface.co/v1/production/uploads/66b711f9512dac2ac08bc5e5/n2kSqNhg-TE56iN_V0xHm.png
Xuanyu Chu
user
CXY07
68308e2697d9a81c8521bc6c
Qianrui Yang
claimed_verified
2025-06-11T16:02:37.345Z
6710e5e68d4cadc83e3aff79
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/DR3uymHp4c3uPeLk_Hmjt.png
Qianrui Yang
user
yangqianrui
68308e2697d9a81c8521bc6d
Zhe-Han Mo
claimed_verified
2025-05-26T08:12:52.048Z
6816bd8e0499f6c7c7b89601
https://cdn-avatars.huggingface.co/v1/production/uploads/6816bd8e0499f6c7c7b89601/-dkIPxjOGbdwDZFxhkBMC.jpeg
Zhe-Han Mo
user
Mo-ZheHan
68308e2697d9a81c8521bc6e
Yiqing Shen
68308e2697d9a81c8521bc6f
Pei-lin Li
68308e2697d9a81c8521bc70
Xinjie Lin
68308e2697d9a81c8521bc71
Jinnian Zhang
68308e2697d9a81c8521bc72
Xin-Sheng Chen
68308e2697d9a81c8521bc73
Yi Zhang
claimed_verified
2025-05-26T08:12:54.379Z
63b2efb5922f26a27e76381c
https://cdn-avatars.huggingface.co/v1/production/uploads/63b2efb5922f26a27e76381c/zOQAt_xywiY8eTvvQOrmQ.png
Yi Zhang
user
uyzhang
68308e2697d9a81c8521bc74
Kiyohiro Nakayama
68308e2697d9a81c8521bc75
Zhengyang Geng
68308e2697d9a81c8521bc76
Houwen Peng
68308e2697d9a81c8521bc77
Han Hu
68308e2697d9a81c8521bc78
Shi-Nin Hu
RBench-V: A Primary Assessment for Visual Reasoning Models with
  Multi-modal Outputs
The rapid advancement of native multi-modal models and omni-models,
exemplified by GPT-4o, Gemini, and o3, with their capability to process and
generate content across modalities such as text and images, marks a significant
milestone in the evolution of intelligence. Systematic evaluation of their
multi-modal output capabilities in visual thinking processes (also known as
multi-modal chain of thought, M-CoT) becomes critically important. However,
existing benchmarks for evaluating multi-modal models primarily focus on
assessing multi-modal inputs and text-only reasoning while neglecting the
importance of reasoning through multi-modal outputs. In this paper, we present
a benchmark, dubbed RBench-V, designed to assess models' vision-indispensable
reasoning abilities. To construct RBench-V, we carefully hand-pick 803
questions covering math, physics, counting, and games. Unlike previous
benchmarks that typically specify certain input modalities, RBench-V presents
problems centered on multi-modal outputs, which require image manipulation such
as generating novel images and constructing auxiliary lines to support the
reasoning process. We evaluate numerous open- and closed-source models on
RBench-V, including o3, Gemini 2.5 Pro, Qwen2.5-VL, etc. Even the
best-performing model, o3, achieves only 25.8% accuracy on RBench-V, far below
the human score of 82.3%, highlighting that current models struggle to leverage
multi-modal reasoning. Data and code are available at
https://evalmodels.github.io/rbenchv
A benchmark called RBench-V evaluates multi-modal models' vision-indispensable reasoning through image manipulation and auxiliary line construction, demonstrating that current models struggle with multi-modal outputs.
multi-modal models omni-models GPT-4o Gemini o3 multi-modal chain of thought M-CoT RBench-V image manipulation auxiliary lines