VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
Vision language models (VLMs) have shown remarkable capabilities in
integrating linguistic and visual reasoning but remain fundamentally limited in
understanding dynamic spatiotemporal interactions. Humans effortlessly track
and reason about object movements, rotations, and perspective shifts-abilities
essential for robust dynamic real-world understanding yet notably lacking in
current VLMs. In this paper, we introduce VLM4D, the first benchmark
specifically designed to evaluate the spatiotemporal reasoning capabilities of
VLMs. Our benchmark comprises diverse real-world and synthetic videos
accompanied by carefully curated question-answer pairs emphasizing
translational and rotational motions, perspective awareness, and motion
continuity. Through comprehensive evaluations of state-of-the-art open and
closed-source VLMs, we identify significant performance gaps compared to human
baselines, highlighting fundamental deficiencies in existing models. Extensive
analysis reveals that VLMs struggle particularly with integrating multiple
visual cues and maintaining temporal coherence. We further explore promising
directions, such as leveraging 4D feature field reconstruction and targeted
spatiotemporal supervised fine-tuning, demonstrating their effectiveness in
enhancing spatiotemporal comprehension. Our work aims to encourage deeper
exploration into improving VLMs' spatial and temporal grounding, paving the way
towards more capable and reliable visual intelligence for dynamic environments.
689a8e78fab6fdd2e52ac3f6
Shijie Zhou
claimed_verified
2025-08-13T07:21:19.330Z
642a276516d4d8293c9a47e8
/avatars/80e6db8bc2544f3486b11b57858a8692.svg
Shijie Zhou
user
shijiezhou
689a8e78fab6fdd2e52ac3f7
Alexander Vilesov
689a8e78fab6fdd2e52ac3f8
Xuehai He
689a8e78fab6fdd2e52ac3f9
Ziyu Wan
689a8e78fab6fdd2e52ac3fa
Shuwang Zhang
689a8e78fab6fdd2e52ac3fb
Aditya Nagachandra
689a8e78fab6fdd2e52ac3fc
Di Chang
689a8e78fab6fdd2e52ac3fd
Dongdong Chen
689a8e78fab6fdd2e52ac3fe
Xin Eric Wang
689a8e78fab6fdd2e52ac3ff
Achuta Kadambi
VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
Vision language models (VLMs) have shown remarkable capabilities in
integrating linguistic and visual reasoning but remain fundamentally limited in
understanding dynamic spatiotemporal interactions. Humans effortlessly track
and reason about object movements, rotations, and perspective shifts-abilities
essential for robust dynamic real-world understanding yet notably lacking in
current VLMs. In this paper, we introduce VLM4D, the first benchmark
specifically designed to evaluate the spatiotemporal reasoning capabilities of
VLMs. Our benchmark comprises diverse real-world and synthetic videos
accompanied by carefully curated question-answer pairs emphasizing
translational and rotational motions, perspective awareness, and motion
continuity. Through comprehensive evaluations of state-of-the-art open and
closed-source VLMs, we identify significant performance gaps compared to human
baselines, highlighting fundamental deficiencies in existing models. Extensive
analysis reveals that VLMs struggle particularly with integrating multiple
visual cues and maintaining temporal coherence. We further explore promising
directions, such as leveraging 4D feature field reconstruction and targeted
spatiotemporal supervised fine-tuning, demonstrating their effectiveness in
enhancing spatiotemporal comprehension. Our work aims to encourage deeper
exploration into improving VLMs' spatial and temporal grounding, paving the way
towards more capable and reliable visual intelligence for dynamic environments.
A benchmark evaluates VLMs' spatiotemporal reasoning, identifying gaps and suggesting improvements like 4D feature field reconstruction and fine-tuning.
VLMs spatiotemporal reasoning VLM4D real-world videos synthetic videos question-answer pairs translational motions rotational motions perspective awareness motion continuity state-of-the-art VLMs human baselines visual cues temporal coherence 4D feature field reconstruction spatiotemporal supervised fine-tuning