Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric
  Videos
Volumetric video represents a transformative advancement in visual media,
enabling users to freely navigate immersive virtual experiences and narrowing
the gap between digital and real worlds. However, the need for extensive manual
intervention to stabilize mesh sequences and the generation of excessively
large assets in existing workflows impedes broader adoption. In this paper, we
present a novel Gaussian-based approach, dubbed DualGS, for real-time
and high-fidelity playback of complex human performance with excellent
compression ratios. Our key idea in DualGS is to separately represent motion
and appearance using the corresponding skin and joint Gaussians. Such an
explicit disentanglement can significantly reduce motion redundancy and enhance
temporal coherence. We begin by initializing the DualGS and anchoring skin
Gaussians to joint Gaussians at the first frame. Subsequently, we employ a
coarse-to-fine training strategy for frame-by-frame human performance modeling.
It includes a coarse alignment phase for overall motion prediction as well as a
fine-grained optimization for robust tracking and high-fidelity rendering. To
integrate volumetric video seamlessly into VR environments, we efficiently
compress motion using entropy encoding and appearance using codec compression
coupled with a persistent codebook. Our approach achieves a compression ratio
of up to 120 times, only requiring approximately 350KB of storage per frame. We
demonstrate the efficacy of our representation through photo-realistic,
free-view experiences on VR headsets, enabling users to immersively watch
musicians in performance and feel the rhythm of the notes at the performers'
fingertips.
66e79c7195bb7553ed7600f4
Yuheng Jiang
66e79c7195bb7553ed7600f5
Zhehao Shen
66e79c7195bb7553ed7600f6
Yu Hong
66e79c7195bb7553ed7600f7
Chengcheng Guo
admin_assigned
2024-09-16T07:36:08.414Z
64e4a9e6b78bc92221b2909e
/avatars/b57b879d43fafa457cbad77dc4d387b1.svg
fy
user
chengchengguo123
66e79c7195bb7553ed7600f8
Yize Wu
66e79c7195bb7553ed7600f9
Yingliang Zhang
66e79c7195bb7553ed7600fa
Jingyi Yu
admin_assigned
2024-09-16T07:34:41.916Z
63ffcec6e7767a89533b8f80
/avatars/b9e8673943741c2e7b80b15a061bb951.svg
JINGYI Yu
user
Daluuu
66e79c7195bb7553ed7600fb
Lan Xu
Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric
  Videos
Volumetric video represents a transformative advancement in visual media,
enabling users to freely navigate immersive virtual experiences and narrowing
the gap between digital and real worlds. However, the need for extensive manual
intervention to stabilize mesh sequences and the generation of excessively
large assets in existing workflows impedes broader adoption. In this paper, we
present a novel Gaussian-based approach, dubbed DualGS, for real-time
and high-fidelity playback of complex human performance with excellent
compression ratios. Our key idea in DualGS is to separately represent motion
and appearance using the corresponding skin and joint Gaussians. Such an
explicit disentanglement can significantly reduce motion redundancy and enhance
temporal coherence. We begin by initializing the DualGS and anchoring skin
Gaussians to joint Gaussians at the first frame. Subsequently, we employ a
coarse-to-fine training strategy for frame-by-frame human performance modeling.
It includes a coarse alignment phase for overall motion prediction as well as a
fine-grained optimization for robust tracking and high-fidelity rendering. To
integrate volumetric video seamlessly into VR environments, we efficiently
compress motion using entropy encoding and appearance using codec compression
coupled with a persistent codebook. Our approach achieves a compression ratio
of up to 120 times, only requiring approximately 350KB of storage per frame. We
demonstrate the efficacy of our representation through photo-realistic,
free-view experiences on VR headsets, enabling users to immersively watch
musicians in performance and feel the rhythm of the notes at the performers'
fingertips.
A Gaussian-based method, DualGS, enables efficient real-time and high-fidelity volumetric video playback with low storage requirements, facilitating immersive VR experiences.
Gaussian-based approach DualGS volumetric video real-time playback high-fidelity compression ratios motion and appearance separation skin and joint Gaussians explicit disentanglement temporal coherence coarse-to-fine training entropy encoding codec compression persistent codebook VR environments photo-realistic free-view experiences