3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming
  of Photo-Realistic Free-Viewpoint Videos
Constructing photo-realistic Free-Viewpoint Videos (FVVs) of dynamic scenes
from multi-view videos remains a challenging endeavor. Despite the remarkable
advancements achieved by current neural rendering techniques, these methods
generally require complete video sequences for offline training and are not
capable of real-time rendering. To address these constraints, we introduce
3DGStream, a method designed for efficient FVV streaming of real-world dynamic
scenes. Our method achieves fast on-the-fly per-frame reconstruction within 12
seconds and real-time rendering at 200 FPS. Specifically, we utilize 3D
Gaussians (3DGs) to represent the scene. Instead of the na\"ive approach of
directly optimizing 3DGs per-frame, we employ a compact Neural Transformation
Cache (NTC) to model the translations and rotations of 3DGs, markedly reducing
the training time and storage required for each FVV frame. Furthermore, we
propose an adaptive 3DG addition strategy to handle emerging objects in dynamic
scenes. Experiments demonstrate that 3DGStream achieves competitive performance
in terms of rendering speed, image quality, training time, and model storage
when compared with state-of-the-art methods.
65e74b52bb169c92fb66e8d4
Jiakai Sun
claimed_verified
2024-03-05T20:40:22.729Z
64744432e0b188d3cb1f6790
/avatars/01c4d7d4724170daba3df60467713345.svg
Jac Sun
user
SJoJoK
65e74b52bb169c92fb66e8d5
Han Jiao
65e74b52bb169c92fb66e8d6
Guangyuan Li
admin_assigned
2024-03-05T20:58:07.919Z
655198a01e13b38aecdf2db4
/avatars/f1bced0e2f4cac2c56ca0dff786c762a.svg
liguangyuan
user
liguangyuan
65e74b52bb169c92fb66e8d7
Zhanjie Zhang
admin_assigned
2024-03-05T20:58:15.475Z
640496240ab5e22719f31719
/avatars/be957a1b0e1c037f03f1439d6142e9ce.svg
Zhanjie Zhang
user
zhangzhanjay
65e74b52bb169c92fb66e8d8
Lei Zhao
65e74b52bb169c92fb66e8d9
Wei Xing
3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming
  of Photo-Realistic Free-Viewpoint Videos
Constructing photo-realistic Free-Viewpoint Videos (FVVs) of dynamic scenes
from multi-view videos remains a challenging endeavor. Despite the remarkable
advancements achieved by current neural rendering techniques, these methods
generally require complete video sequences for offline training and are not
capable of real-time rendering. To address these constraints, we introduce
3DGStream, a method designed for efficient FVV streaming of real-world dynamic
scenes. Our method achieves fast on-the-fly per-frame reconstruction within 12
seconds and real-time rendering at 200 FPS. Specifically, we utilize 3D
Gaussians (3DGs) to represent the scene. Instead of the na\"ive approach of
directly optimizing 3DGs per-frame, we employ a compact Neural Transformation
Cache (NTC) to model the translations and rotations of 3DGs, markedly reducing
the training time and storage required for each FVV frame. Furthermore, we
propose an adaptive 3DG addition strategy to handle emerging objects in dynamic
scenes. Experiments demonstrate that 3DGStream achieves competitive performance
in terms of rendering speed, image quality, training time, and model storage
when compared with state-of-the-art methods.
3DGStream efficiently streams photo-realistic Free-Viewpoint Videos in real-time using 3D Gaussians and a Neural Transformation Cache, competitive in speed, quality, training time, and storage.
3DGaussians Neural Transformation Cache adaptive 3DG addition strategy Free-Viewpoint Videos real-time rendering