Streaming 4D Visual Geometry Transformer
Perceiving and reconstructing 4D spatial-temporal geometry from videos is a
fundamental yet challenging computer vision task. To facilitate interactive and
real-time applications, we propose a streaming 4D visual geometry transformer
that shares a similar philosophy with autoregressive large language models. We
explore a simple and efficient design and employ a causal transformer
architecture to process the input sequence in an online manner. We use temporal
causal attention and cache the historical keys and values as implicit memory to
enable efficient streaming long-term 4D reconstruction. This design can handle
real-time 4D reconstruction by incrementally integrating historical information
while maintaining high-quality spatial consistency. For efficient training, we
propose to distill knowledge from the dense bidirectional visual geometry
grounded transformer (VGGT) to our causal model. For inference, our model
supports the migration of optimized efficient attention operator (e.g.,
FlashAttention) from the field of large language models. Extensive experiments
on various 4D geometry perception benchmarks demonstrate that our model
increases the inference speed in online scenarios while maintaining competitive
performance, paving the way for scalable and interactive 4D vision systems.
Code is available at: https://github.com/wzzheng/StreamVGGT.
68780501001546c83aa4f913
Dong Zhuo
claimed_verified
2025-07-17T08:22:30.731Z
65716c354f98a7a5a3e0bd40
/avatars/f7c4fe32ae64d17a7b5b46c7de82be37.svg
paryi
user
paryi
68780501001546c83aa4f914
Wenzhao Zheng
68780501001546c83aa4f915
Jiahe Guo
claimed_verified
2025-07-18T07:50:38.396Z
67583544b7759e72b80853b3
/avatars/2fe352413eca4d3a1d506a021e863312.svg
Guo JiaHe
user
lch01
68780501001546c83aa4f916
Yuqi Wu
68780501001546c83aa4f917
Jie Zhou
68780501001546c83aa4f918
Jiwen Lu
Streaming 4D Visual Geometry Transformer
Perceiving and reconstructing 4D spatial-temporal geometry from videos is a
fundamental yet challenging computer vision task. To facilitate interactive and
real-time applications, we propose a streaming 4D visual geometry transformer
that shares a similar philosophy with autoregressive large language models. We
explore a simple and efficient design and employ a causal transformer
architecture to process the input sequence in an online manner. We use temporal
causal attention and cache the historical keys and values as implicit memory to
enable efficient streaming long-term 4D reconstruction. This design can handle
real-time 4D reconstruction by incrementally integrating historical information
while maintaining high-quality spatial consistency. For efficient training, we
propose to distill knowledge from the dense bidirectional visual geometry
grounded transformer (VGGT) to our causal model. For inference, our model
supports the migration of optimized efficient attention operator (e.g.,
FlashAttention) from the field of large language models. Extensive experiments
on various 4D geometry perception benchmarks demonstrate that our model
increases the inference speed in online scenarios while maintaining competitive
performance, paving the way for scalable and interactive 4D vision systems.
Code is available at: https://github.com/wzzheng/StreamVGGT.
A streaming 4D visual geometry transformer uses causal attention and knowledge distillation to achieve real-time 4D reconstruction with high spatial consistency and competitive performance.
4D spatial-temporal geometry streaming 4D visual geometry transformer causal transformer architecture temporal causal attention implicit memory real-time 4D reconstruction spatial consistency knowledge distillation dense bidirectional visual geometry grounded transformer (VGGT) FlashAttention 4D geometry perception benchmarks