4Real-Video-V2: Fused View-Time Attention and Feedforward Reconstruction
  for 4D Scene Generation
We propose the first framework capable of computing a 4D spatio-temporal grid
of video frames and 3D Gaussian particles for each time step using a
feed-forward architecture. Our architecture has two main components, a 4D video
model and a 4D reconstruction model. In the first part, we analyze current 4D
video diffusion architectures that perform spatial and temporal attention
either sequentially or in parallel within a two-stream design. We highlight the
limitations of existing approaches and introduce a novel fused architecture
that performs spatial and temporal attention within a single layer. The key to
our method is a sparse attention pattern, where tokens attend to others in the
same frame, at the same timestamp, or from the same viewpoint. In the second
part, we extend existing 3D reconstruction algorithms by introducing a Gaussian
head, a camera token replacement algorithm, and additional dynamic layers and
training. Overall, we establish a new state of the art for 4D generation,
improving both visual quality and reconstruction capability.
685aa5810e4ad7e219758779
Chaoyang Wang
685aa5810e4ad7e21975877a
Ashkan Mirzaei
685aa5810e4ad7e21975877b
Vidit Goel
685aa5810e4ad7e21975877c
Willi Menapace
685aa5810e4ad7e21975877d
Aliaksandr Siarohin
685aa5810e4ad7e21975877e
Avalon Vinella
685aa5810e4ad7e21975877f
Michael Vasilkovsky
685aa5810e4ad7e219758780
Ivan Skorokhodov
685aa5810e4ad7e219758781
Vladislav Shakhrai
685aa5810e4ad7e219758782
Sergey Korolev
685aa5810e4ad7e219758783
Sergey Tulyakov
685aa5810e4ad7e219758784
Peter Wonka
4Real-Video-V2: Fused View-Time Attention and Feedforward Reconstruction
  for 4D Scene Generation
We propose the first framework capable of computing a 4D spatio-temporal grid
of video frames and 3D Gaussian particles for each time step using a
feed-forward architecture. Our architecture has two main components, a 4D video
model and a 4D reconstruction model. In the first part, we analyze current 4D
video diffusion architectures that perform spatial and temporal attention
either sequentially or in parallel within a two-stream design. We highlight the
limitations of existing approaches and introduce a novel fused architecture
that performs spatial and temporal attention within a single layer. The key to
our method is a sparse attention pattern, where tokens attend to others in the
same frame, at the same timestamp, or from the same viewpoint. In the second
part, we extend existing 3D reconstruction algorithms by introducing a Gaussian
head, a camera token replacement algorithm, and additional dynamic layers and
training. Overall, we establish a new state of the art for 4D generation,
improving both visual quality and reconstruction capability.
A new framework combines 4D video modeling and 3D reconstruction using a unified architecture with sparse attention patterns, achieving superior visual quality and reconstruction.
4D spatio-temporal grid feed-forward architecture 4D video model 4D reconstruction model 4D video diffusion spatial attention temporal attention fused architecture sparse attention pattern Gaussian head camera token replacement dynamic layers