MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning
The ability to process information from multiple modalities and to reason
through it step-by-step remains a critical challenge in advancing artificial
intelligence. However, existing reasoning benchmarks focus on text-only
reasoning, or employ multimodal questions that can be answered by directly
retrieving information from a non-text modality. Thus, complex reasoning
remains poorly understood in multimodal domains. Here, we present MARBLE, a
challenging multimodal reasoning benchmark that is designed to scrutinize
multimodal language models (MLLMs) in their ability to carefully reason
step-by-step through complex multimodal problems and environments. MARBLE is
composed of two highly challenging tasks, M-Portal and M-Cube, that require the
crafting and understanding of multistep plans under spatial, visual, and
physical constraints. We find that current MLLMs perform poorly on MARBLE --
all the 12 advanced models obtain near-random performance on M-Portal and 0%
accuracy on M-Cube. Only in simplified subtasks some models outperform the
random baseline, indicating that complex reasoning is still a challenge for
existing MLLMs. Moreover, we show that perception remains a bottleneck, where
MLLMs occasionally fail to extract information from the visual inputs. By
shedding a light on the limitations of MLLMs, we hope that MARBLE will spur the
development of the next generation of models with the ability to reason and
plan across many, multimodal reasoning steps.
686370bd588cea0da970c90e
Yulun Jiang
claimed_verified
2025-07-01T10:13:16.902Z
662a75287181150b857245fb
/avatars/6a489bf6d3d9cd26ba88f17b35c6ecb5.svg
Yulun Jiang
user
yljblues
686370bd588cea0da970c90f
Yekun Chai
686370bd588cea0da970c910
Maria Brbić
686370bd588cea0da970c911
Michael Moor
claimed_verified
2025-07-01T10:13:19.043Z
6438d1d843d932c462404500
https://cdn-avatars.huggingface.co/v1/production/uploads/6438d1d843d932c462404500/6UrtJbed9N5ETbImgIh-C.png
Michael Moor
user
mdmoor
MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning
The ability to process information from multiple modalities and to reason
through it step-by-step remains a critical challenge in advancing artificial
intelligence. However, existing reasoning benchmarks focus on text-only
reasoning, or employ multimodal questions that can be answered by directly
retrieving information from a non-text modality. Thus, complex reasoning
remains poorly understood in multimodal domains. Here, we present MARBLE, a
challenging multimodal reasoning benchmark that is designed to scrutinize
multimodal language models (MLLMs) in their ability to carefully reason
step-by-step through complex multimodal problems and environments. MARBLE is
composed of two highly challenging tasks, M-Portal and M-Cube, that require the
crafting and understanding of multistep plans under spatial, visual, and
physical constraints. We find that current MLLMs perform poorly on MARBLE --
all the 12 advanced models obtain near-random performance on M-Portal and 0%
accuracy on M-Cube. Only in simplified subtasks some models outperform the
random baseline, indicating that complex reasoning is still a challenge for
existing MLLMs. Moreover, we show that perception remains a bottleneck, where
MLLMs occasionally fail to extract information from the visual inputs. By
shedding a light on the limitations of MLLMs, we hope that MARBLE will spur the
development of the next generation of models with the ability to reason and
plan across many, multimodal reasoning steps.
MARBLE is a challenging multimodal reasoning benchmark that tests the step-by-step reasoning capabilities of multimodal language models across complex tasks, highlighting their limitations in perception and reasoning.
multimodal reasoning multimodal language models M-Portal M-Cube multistep plans spatial constraints visual constraints physical constraints perception bottleneck