Thinking in Space: How Multimodal Large Language Models See, Remember,
  and Recall Spaces
Humans possess the visual-spatial intelligence to remember spaces from
sequential visual observations. However, can Multimodal Large Language Models
(MLLMs) trained on million-scale video datasets also ``think in space'' from
videos? We present a novel video-based visual-spatial intelligence benchmark
(VSI-Bench) of over 5,000 question-answer pairs, and find that MLLMs exhibit
competitive - though subhuman - visual-spatial intelligence. We probe models to
express how they think in space both linguistically and visually and find that
while spatial reasoning capabilities remain the primary bottleneck for MLLMs to
reach higher benchmark performance, local world models and spatial awareness do
emerge within these models. Notably, prevailing linguistic reasoning techniques
(e.g., chain-of-thought, self-consistency, tree-of-thoughts) fail to improve
performance, whereas explicitly generating cognitive maps during
question-answering enhances MLLMs' spatial distance ability.
676426104fa1553ccc44451f
Jihan Yang
admin_assigned
2024-12-20T09:53:33.378Z
6304baf041387c7f1177a5d2
/avatars/795c63f2394080eec78ca7981d4a1f78.svg
Jihan Yang
user
jihanyang
676426104fa1553ccc444520
Shusheng Yang
claimed_verified
2024-12-19T18:00:36.526Z
627ccf058b4e56cfc2716425
https://cdn-avatars.huggingface.co/v1/production/uploads/1652346592327-noauth.jpeg
Shusheng Yang
user
ShushengYang
676426104fa1553ccc444521
Anjali W. Gupta
claimed_verified
2024-12-21T15:19:58.834Z
654a6c59f8ebcec54510ee56
/avatars/31089658df2d754ab6e4f6ed2750cc1e.svg
Anjali W Gupta
user
anjaliwgupta
676426104fa1553ccc444522
Rilyn Han
admin_assigned
2024-12-20T09:53:14.391Z
657ad3ed2b365884914652c0
https://cdn-avatars.huggingface.co/v1/production/uploads/657ad3ed2b365884914652c0/J86SIaWafJEtqxWH84o0Z.jpeg
Rilyn Han
user
rilynhan
676426104fa1553ccc444523
Li Fei-Fei
676426104fa1553ccc444524
Saining Xie
admin_assigned
2024-12-20T09:52:57.712Z
6596422646624a86ff3b3bda
/avatars/216e12b77e45ac5f1fa20932f5745411.svg
Saining Xie
user
sainx
Thinking in Space: How Multimodal Large Language Models See, Remember,
  and Recall Spaces
Humans possess the visual-spatial intelligence to remember spaces from
sequential visual observations. However, can Multimodal Large Language Models
(MLLMs) trained on million-scale video datasets also ``think in space'' from
videos? We present a novel video-based visual-spatial intelligence benchmark
(VSI-Bench) of over 5,000 question-answer pairs, and find that MLLMs exhibit
competitive - though subhuman - visual-spatial intelligence. We probe models to
express how they think in space both linguistically and visually and find that
while spatial reasoning capabilities remain the primary bottleneck for MLLMs to
reach higher benchmark performance, local world models and spatial awareness do
emerge within these models. Notably, prevailing linguistic reasoning techniques
(e.g., chain-of-thought, self-consistency, tree-of-thoughts) fail to improve
performance, whereas explicitly generating cognitive maps during
question-answering enhances MLLMs' spatial distance ability.
MLLMs trained on large video datasets show competitive but subhuman visual-spatial intelligence, with spatial reasoning as a key bottleneck that can be improved by generating cognitive maps.
Multimodal Large Language Models VSI-Bench visual-spatial intelligence spatial reasoning cognitive maps