STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer
We present STream3R, a novel approach to 3D reconstruction that reformulates
pointmap prediction as a decoder-only Transformer problem. Existing
state-of-the-art methods for multi-view reconstruction either depend on
expensive global optimization or rely on simplistic memory mechanisms that
scale poorly with sequence length. In contrast, STream3R introduces an
streaming framework that processes image sequences efficiently using causal
attention, inspired by advances in modern language modeling. By learning
geometric priors from large-scale 3D datasets, STream3R generalizes well to
diverse and challenging scenarios, including dynamic scenes where traditional
methods often fail. Extensive experiments show that our method consistently
outperforms prior work across both static and dynamic scene benchmarks.
Moreover, STream3R is inherently compatible with LLM-style training
infrastructure, enabling efficient large-scale pretraining and fine-tuning for
various downstream 3D tasks. Our results underscore the potential of causal
Transformer models for online 3D perception, paving the way for real-time 3D
understanding in streaming environments. More details can be found in our
project page: https://nirvanalan.github.io/projects/stream3r.
689efb1da4caabb4320e5e7a
Yushi Lan
689efb1da4caabb4320e5e7b
Yihang Luo
claimed_verified
2025-08-18T06:56:51.343Z
654b607b9296401f072e7cf5
https://cdn-avatars.huggingface.co/v1/production/uploads/654b607b9296401f072e7cf5/5ys396eFk9x_vKt5Aot8G.jpeg
Yihang Luo
user
Luo-Yihang
689efb1da4caabb4320e5e7c
Fangzhou Hong
689efb1da4caabb4320e5e7d
Shangchen Zhou
689efb1da4caabb4320e5e7e
Honghua Chen
689efb1da4caabb4320e5e7f
Zhaoyang Lyu
689efb1da4caabb4320e5e80
Shuai Yang
689efb1da4caabb4320e5e81
Bo Dai
689efb1da4caabb4320e5e82
Chen Change Loy
689efb1da4caabb4320e5e83
Xingang Pan
STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer
We present STream3R, a novel approach to 3D reconstruction that reformulates
pointmap prediction as a decoder-only Transformer problem. Existing
state-of-the-art methods for multi-view reconstruction either depend on
expensive global optimization or rely on simplistic memory mechanisms that
scale poorly with sequence length. In contrast, STream3R introduces an
streaming framework that processes image sequences efficiently using causal
attention, inspired by advances in modern language modeling. By learning
geometric priors from large-scale 3D datasets, STream3R generalizes well to
diverse and challenging scenarios, including dynamic scenes where traditional
methods often fail. Extensive experiments show that our method consistently
outperforms prior work across both static and dynamic scene benchmarks.
Moreover, STream3R is inherently compatible with LLM-style training
infrastructure, enabling efficient large-scale pretraining and fine-tuning for
various downstream 3D tasks. Our results underscore the potential of causal
Transformer models for online 3D perception, paving the way for real-time 3D
understanding in streaming environments. More details can be found in our
project page: https://nirvanalan.github.io/projects/stream3r.
STream3R reformulates 3D reconstruction as a decoder-only Transformer problem, using causal attention to efficiently process image sequences and outperform existing methods in both static and dynamic scenes.
Transformer causal attention 3D reconstruction pointmap prediction multi-view reconstruction global optimization memory mechanisms geometric priors LLM-style training pretraining fine-tuning online 3D perception real-time 3D understanding streaming environments