Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View
  Scenes
Understanding 3D spatial relationships remains a major limitation of current
Vision-Language Models (VLMs). Prior work has addressed this issue by creating
spatial question-answering (QA) datasets based on single images or indoor
videos. However, real-world embodied AI agents such as robots and self-driving
cars typically rely on ego-centric, multi-view observations. To this end, we
introduce Ego3D-Bench, a new benchmark designed to evaluate the spatial
reasoning abilities of VLMs using ego-centric, multi-view outdoor data.
Ego3D-Bench comprises over 8,600 QA pairs, created with significant involvement
from human annotators to ensure quality and diversity. We benchmark 16 SOTA
VLMs, including GPT-4o, Gemini1.5-Pro, InternVL3, and Qwen2.5-VL. Our results
reveal a notable performance gap between human level scores and VLM
performance, highlighting that current VLMs still fall short of human level
spatial understanding. To bridge this gap, we propose Ego3D-VLM, a
post-training framework that enhances 3D spatial reasoning of VLMs. Ego3D-VLM
generates cognitive map based on estimated global 3D coordinates, resulting in
12% average improvement on multi-choice QA and 56% average improvement on
absolute distance estimation. Ego3D-VLM is modular and can be integrated with
any existing VLM. Together, Ego3D-Bench and Ego3D-VLM offer valuable tools for
advancing toward human level spatial understanding in real-world, multi-view
environments.
68c1eab629b8ec9932cd0866
Mohsen Gholami
claimed_verified
2025-09-12T16:15:48.569Z
64ed16472f0bd58125027ff1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/H59YRVHXb4EUTMK5rX47k.png
Mohsen Gholami
user
mgholami
68c1eab629b8ec9932cd0867
Ahmad Rezaei
claimed_verified
2025-09-13T14:56:40.152Z
6495d9b6e6692d3676406834
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/W7ktMxTNo75rqUtTWJfCw.png
Ahmad Rezaei
user
AhNr
68c1eab629b8ec9932cd0868
Zhou Weimin
68c1eab629b8ec9932cd0869
Yong Zhang
68c1eab629b8ec9932cd086a
Mohammad Akbari
claimed_verified
2025-09-12T16:15:45.430Z
6545976cb8ac1a89ffa8d6cb
/avatars/4678cad6e65e42ef67f517e9c669031d.svg
Mohammad Akbari
user
moak7
Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View
  Scenes
Understanding 3D spatial relationships remains a major limitation of current
Vision-Language Models (VLMs). Prior work has addressed this issue by creating
spatial question-answering (QA) datasets based on single images or indoor
videos. However, real-world embodied AI agents such as robots and self-driving
cars typically rely on ego-centric, multi-view observations. To this end, we
introduce Ego3D-Bench, a new benchmark designed to evaluate the spatial
reasoning abilities of VLMs using ego-centric, multi-view outdoor data.
Ego3D-Bench comprises over 8,600 QA pairs, created with significant involvement
from human annotators to ensure quality and diversity. We benchmark 16 SOTA
VLMs, including GPT-4o, Gemini1.5-Pro, InternVL3, and Qwen2.5-VL. Our results
reveal a notable performance gap between human level scores and VLM
performance, highlighting that current VLMs still fall short of human level
spatial understanding. To bridge this gap, we propose Ego3D-VLM, a
post-training framework that enhances 3D spatial reasoning of VLMs. Ego3D-VLM
generates cognitive map based on estimated global 3D coordinates, resulting in
12% average improvement on multi-choice QA and 56% average improvement on
absolute distance estimation. Ego3D-VLM is modular and can be integrated with
any existing VLM. Together, Ego3D-Bench and Ego3D-VLM offer valuable tools for
advancing toward human level spatial understanding in real-world, multi-view
environments.
Ego3D-Bench evaluates VLMs on ego-centric, multi-view outdoor data, revealing performance gaps, and Ego3D-VLM enhances 3D spatial reasoning through cognitive map generation.
Vision-Language Models spatial question-answering ego-centric multi-view cognitive map global 3D coordinates multi-choice QA absolute distance estimation