LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?
Multi-step spatial reasoning entails understanding and reasoning about
spatial relationships across multiple sequential steps, which is crucial for
tackling complex real-world applications, such as robotic manipulation,
autonomous navigation, and automated assembly. To assess how well current
Multimodal Large Language Models (MLLMs) have acquired this fundamental
capability, we introduce LEGO-Puzzles, a scalable benchmark designed
to evaluate both spatial understanding and sequential
reasoning in MLLMs through LEGO-based tasks. LEGO-Puzzles consists of 1,100
carefully curated visual question-answering (VQA) samples spanning 11 distinct
tasks, ranging from basic spatial understanding to complex multi-step
reasoning. Based on LEGO-Puzzles, we conduct a comprehensive evaluation of
state-of-the-art MLLMs and uncover significant limitations in their spatial
reasoning capabilities: even the most powerful MLLMs can answer only about half
of the test cases, whereas human participants achieve over 90\% accuracy. In
addition to VQA tasks, we evaluate MLLMs' abilities to generate LEGO images
following assembly illustrations. Our experiments show that only
Gemini-2.0-Flash and GPT-4o exhibit a limited ability to follow these
instructions, while other MLLMs either replicate the input image or generate
completely irrelevant outputs. Overall, LEGO-Puzzles exposes critical
deficiencies in existing MLLMs' spatial understanding and sequential reasoning
capabilities, and underscores the need for further advancements in multimodal
spatial reasoning.
67e4d3df7e97884ba4150ec0
Kexian Tang
admin_assigned
2025-03-27T09:46:12.696Z
662516d72419feed62fb3a0a
/avatars/24c4157829e70a4e346aa984885aa5ad.svg
Kexian Tang
user
KexianTang
67e4d3df7e97884ba4150ec1
Junyao Gao
claimed_verified
2025-03-27T09:03:15.061Z
64a6ae0e0437599198cf3a98
/avatars/6635432cc0589ba12dc170cad6986d6d.svg
Junyao Gao
user
favourisnotyou
67e4d3df7e97884ba4150ec2
Yanhong Zeng
admin_assigned
2025-03-27T09:46:24.775Z
63d4b843df01ef426a0f79fb
https://cdn-avatars.huggingface.co/v1/production/uploads/1676365795587-63d4b843df01ef426a0f79fb.jpeg
Yanhong Zeng
user
zengyh1900
67e4d3df7e97884ba4150ec3
Haodong Duan
claimed_verified
2025-03-27T09:00:33.733Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
67e4d3df7e97884ba4150ec4
Yanan Sun
67e4d3df7e97884ba4150ec5
Zhening Xing
claimed_verified
2025-03-27T09:00:36.341Z
62fb2a9dc95d426ff8f74c8d
/avatars/25c1a68ee7b7d0cc7e9f56bde37f4914.svg
Zhening Xing
user
Leoxing
67e4d3df7e97884ba4150ec6
Wenran Liu
admin_assigned
2025-03-27T09:46:45.866Z
6385f8598b5acae8d24caf16
/avatars/9d261f95d24e882157b987b8827098be.svg
liuwenran
user
lwrshi1965
67e4d3df7e97884ba4150ec7
Kaifeng Lyu
admin_assigned
2025-03-27T09:46:52.341Z
6414230a0fcefcf72e5085dd
/avatars/3a38dc8c84b0f27af846184d1c19f6ef.svg
Kaifeng Lyu
user
vfleaking
67e4d3df7e97884ba4150ec8
Kai Chen
LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?
Multi-step spatial reasoning entails understanding and reasoning about
spatial relationships across multiple sequential steps, which is crucial for
tackling complex real-world applications, such as robotic manipulation,
autonomous navigation, and automated assembly. To assess how well current
Multimodal Large Language Models (MLLMs) have acquired this fundamental
capability, we introduce LEGO-Puzzles, a scalable benchmark designed
to evaluate both spatial understanding and sequential
reasoning in MLLMs through LEGO-based tasks. LEGO-Puzzles consists of 1,100
carefully curated visual question-answering (VQA) samples spanning 11 distinct
tasks, ranging from basic spatial understanding to complex multi-step
reasoning. Based on LEGO-Puzzles, we conduct a comprehensive evaluation of
state-of-the-art MLLMs and uncover significant limitations in their spatial
reasoning capabilities: even the most powerful MLLMs can answer only about half
of the test cases, whereas human participants achieve over 90\% accuracy. In
addition to VQA tasks, we evaluate MLLMs' abilities to generate LEGO images
following assembly illustrations. Our experiments show that only
Gemini-2.0-Flash and GPT-4o exhibit a limited ability to follow these
instructions, while other MLLMs either replicate the input image or generate
completely irrelevant outputs. Overall, LEGO-Puzzles exposes critical
deficiencies in existing MLLMs' spatial understanding and sequential reasoning
capabilities, and underscores the need for further advancements in multimodal
spatial reasoning.
LEGO-Puzzles evaluates the spatial understanding and sequential reasoning capabilities of Multimodal Large Language Models (MLLMs) through LEGO-based tasks, revealing significant limitations compared to human performance.
spatial reasoning sequential reasoning Multimodal Large Language Models (MLLMs) LEGO-Puzzles visual question-answering (VQA) spatial understanding Gemini-2.0-Flash GPT-4o