Video World Models with Long-term Spatial Memory
Emerging world models autoregressively generate video frames in response to
actions, such as camera movements and text prompts, among other control
signals. Due to limited temporal context window sizes, these models often
struggle to maintain scene consistency during revisits, leading to severe
forgetting of previously generated environments. Inspired by the mechanisms of
human memory, we introduce a novel framework to enhancing long-term consistency
of video world models through a geometry-grounded long-term spatial memory. Our
framework includes mechanisms to store and retrieve information from the
long-term spatial memory and we curate custom datasets to train and evaluate
world models with explicitly stored 3D memory mechanisms. Our evaluations show
improved quality, consistency, and context length compared to relevant
baselines, paving the way towards long-term consistent world generation.
6842929c46106f29d78635ad
Tong Wu
6842929c46106f29d78635ae
Shuai Yang
claimed_verified
2025-06-07T05:48:43.406Z
6373037cd3ba6bd3f9bc32fa
https://cdn-avatars.huggingface.co/v1/production/uploads/6373037cd3ba6bd3f9bc32fa/dIL4AIcL24QL_Rp4B9ewS.png
Shuai Yang
user
ysmikey
6842929c46106f29d78635af
Ryan Po
6842929c46106f29d78635b0
Yinghao Xu
6842929c46106f29d78635b1
Ziwei Liu
6842929c46106f29d78635b2
Dahua Lin
6842929c46106f29d78635b3
Gordon Wetzstein
Video World Models with Long-term Spatial Memory
Emerging world models autoregressively generate video frames in response to
actions, such as camera movements and text prompts, among other control
signals. Due to limited temporal context window sizes, these models often
struggle to maintain scene consistency during revisits, leading to severe
forgetting of previously generated environments. Inspired by the mechanisms of
human memory, we introduce a novel framework to enhancing long-term consistency
of video world models through a geometry-grounded long-term spatial memory. Our
framework includes mechanisms to store and retrieve information from the
long-term spatial memory and we curate custom datasets to train and evaluate
world models with explicitly stored 3D memory mechanisms. Our evaluations show
improved quality, consistency, and context length compared to relevant
baselines, paving the way towards long-term consistent world generation.
A new framework enhances video world models' long-term consistency by integrating a geometry-grounded long-term spatial memory mechanism.
world models autoregressive generation video frames control signals temporal context window scene consistency long-term spatial memory custom datasets 3D memory mechanisms