OST-Bench: Evaluating the Capabilities of MLLMs in Online
  Spatio-temporal Scene Understanding
Recent advances in multimodal large language models (MLLMs) have shown
remarkable capabilities in integrating vision and language for complex
reasoning. While most existing benchmarks evaluate models under offline
settings with a fixed set of pre-recorded inputs, we introduce OST-Bench, a
benchmark designed to evaluate Online Spatio-Temporal understanding from the
perspective of an agent actively exploring a scene. The Online aspect
emphasizes the need to process and reason over incrementally acquired
observations, while the Spatio-Temporal component requires integrating current
visual inputs with historical memory to support dynamic spatial reasoning.
OST-Bench better reflects the challenges of real-world embodied perception.
Built on an efficient data collection pipeline, OST-Bench consists of 1.4k
scenes and 10k question-answer pairs collected from ScanNet, Matterport3D, and
ARKitScenes. We evaluate several leading MLLMs on OST-Bench and observe that
they fall short on tasks requiring complex spatio-temporal reasoning. Under the
online setting, their accuracy declines as the exploration horizon extends and
the memory grows. Through further experimental analysis, we identify common
error patterns across models and find that both complex clue-based spatial
reasoning demands and long-term memory retrieval requirements significantly
drop model performance along two separate axes, highlighting the core
challenges that must be addressed to improve online embodied reasoning. To
foster further research and development in the field, our codes, dataset, and
benchmark are available. Our project page is:
https://rbler1234.github.io/OSTBench.github.io/
687088a6c8391850d6097874
JingLi Lin
687088a6c8391850d6097875
Chenming Zhu
claimed_verified
2025-07-11T13:12:39.911Z
6433aba4546e16f17a0f19f6
https://cdn-avatars.huggingface.co/v1/production/uploads/6433aba4546e16f17a0f19f6/3hlNy4_Suy0bO__Qa9amL.jpeg
Chenming Zhu
user
ChaimZhu
687088a6c8391850d6097876
Runsen Xu
687088a6c8391850d6097877
Xiaohan Mao
687088a6c8391850d6097878
Xihui Liu
687088a6c8391850d6097879
Tai Wang
687088a6c8391850d609787a
Jiangmiao Pang
OST-Bench: Evaluating the Capabilities of MLLMs in Online
  Spatio-temporal Scene Understanding
Recent advances in multimodal large language models (MLLMs) have shown
remarkable capabilities in integrating vision and language for complex
reasoning. While most existing benchmarks evaluate models under offline
settings with a fixed set of pre-recorded inputs, we introduce OST-Bench, a
benchmark designed to evaluate Online Spatio-Temporal understanding from the
perspective of an agent actively exploring a scene. The Online aspect
emphasizes the need to process and reason over incrementally acquired
observations, while the Spatio-Temporal component requires integrating current
visual inputs with historical memory to support dynamic spatial reasoning.
OST-Bench better reflects the challenges of real-world embodied perception.
Built on an efficient data collection pipeline, OST-Bench consists of 1.4k
scenes and 10k question-answer pairs collected from ScanNet, Matterport3D, and
ARKitScenes. We evaluate several leading MLLMs on OST-Bench and observe that
they fall short on tasks requiring complex spatio-temporal reasoning. Under the
online setting, their accuracy declines as the exploration horizon extends and
the memory grows. Through further experimental analysis, we identify common
error patterns across models and find that both complex clue-based spatial
reasoning demands and long-term memory retrieval requirements significantly
drop model performance along two separate axes, highlighting the core
challenges that must be addressed to improve online embodied reasoning. To
foster further research and development in the field, our codes, dataset, and
benchmark are available. Our project page is:
https://rbler1234.github.io/OSTBench.github.io/
OST-Bench evaluates multimodal large language models on online spatio-temporal reasoning tasks, highlighting challenges in dynamic spatial understanding and long-term memory retrieval.
multimodal large language models MLLMs OST-Bench Online Spatio-Temporal understanding ScanNet Matterport3D ARKitScenes spatio-temporal reasoning long-term memory retrieval