VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed
  View Memory
We propose a novel memory mechanism to build video generators that can
explore environments interactively. Similar results have previously been
achieved by out-painting 2D views of the scene while incrementally
reconstructing its 3D geometry, which quickly accumulates errors, or by video
generators with a short context window, which struggle to maintain scene
coherence over the long term. To address these limitations, we introduce
Surfel-Indexed View Memory (VMem), a mechanism that remembers past views by
indexing them geometrically based on the 3D surface elements (surfels) they
have observed. VMem enables the efficient retrieval of the most relevant past
views when generating new ones. By focusing only on these relevant views, our
method produces consistent explorations of imagined environments at a fraction
of the computational cost of using all past views as context. We evaluate our
approach on challenging long-term scene synthesis benchmarks and demonstrate
superior performance compared to existing methods in maintaining scene
coherence and camera control.
685a1ce80e4ad7e2197585b7
Runjia Li
admin_assigned
2025-06-24T12:54:50.758Z
638e29cf319f9c746b87ad4b
/avatars/70cac8d47847c389eb0393051a64c4a4.svg
Runjia Li
user
liguang0115
685a1ce80e4ad7e2197585b8
Philip Torr
admin_assigned
2025-06-24T12:54:41.794Z
6565ed28a5ec0231cb07225f
/avatars/7f95bba9aa7811d56eecb380827abfac.svg
prof philip torr
user
philiptorr
685a1ce80e4ad7e2197585b9
Andrea Vedaldi
admin_assigned
2025-06-24T12:54:34.616Z
68055d1082c2bae6af503848
/avatars/e32c2f0a8efbf54707ead520ae9374bf.svg
Andrea Vedaldi
user
vedaldi
685a1ce80e4ad7e2197585ba
Tomas Jakab
admin_assigned
2025-06-24T12:54:25.185Z
6435823f76dbfd731bca13ab
/avatars/67cd864bb8b1310a344e60a27b1ed075.svg
Tomas Jakab
user
tmsj
VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed
  View Memory
We propose a novel memory mechanism to build video generators that can
explore environments interactively. Similar results have previously been
achieved by out-painting 2D views of the scene while incrementally
reconstructing its 3D geometry, which quickly accumulates errors, or by video
generators with a short context window, which struggle to maintain scene
coherence over the long term. To address these limitations, we introduce
Surfel-Indexed View Memory (VMem), a mechanism that remembers past views by
indexing them geometrically based on the 3D surface elements (surfels) they
have observed. VMem enables the efficient retrieval of the most relevant past
views when generating new ones. By focusing only on these relevant views, our
method produces consistent explorations of imagined environments at a fraction
of the computational cost of using all past views as context. We evaluate our
approach on challenging long-term scene synthesis benchmarks and demonstrate
superior performance compared to existing methods in maintaining scene
coherence and camera control.
A novel memory mechanism called Surfel-Indexed View Memory enhances video generation by efficiently remembering and retrieving relevant past views, improving long-term scene coherence and reducing computational cost.
memory mechanism video generators out-painting 3D geometry context window Surfel-Indexed View Memory surfels scene coherence camera control long-term scene synthesis benchmarks