HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian
  Splatting
We have recently seen tremendous progress in photo-real human modeling and
rendering. Yet, efficiently rendering realistic human performance and
integrating it into the rasterization pipeline remains challenging. In this
paper, we present HiFi4G, an explicit and compact Gaussian-based approach for
high-fidelity human performance rendering from dense footage. Our core
intuition is to marry the 3D Gaussian representation with non-rigid tracking,
achieving a compact and compression-friendly representation. We first propose a
dual-graph mechanism to obtain motion priors, with a coarse deformation graph
for effective initialization and a fine-grained Gaussian graph to enforce
subsequent constraints. Then, we utilize a 4D Gaussian optimization scheme with
adaptive spatial-temporal regularizers to effectively balance the non-rigid
prior and Gaussian updating. We also present a companion compression scheme
with residual compensation for immersive experiences on various platforms. It
achieves a substantial compression rate of approximately 25 times, with less
than 2MB of storage per frame. Extensive experiments demonstrate the
effectiveness of our approach, which significantly outperforms existing
approaches in terms of optimization speed, rendering quality, and storage
overhead.
65713bdf87a92b76924ecd4d
Yuheng Jiang
65713bdf87a92b76924ecd4e
Zhehao Shen
65713bdf87a92b76924ecd4f
Penghao Wang
admin_assigned
2023-12-07T11:56:28.616Z
654a064a8ceb5deea8c53acb
/avatars/b6fc5c4f62c75b0866926a0da541620f.svg
Penghao Wang
user
phw00007
65713bdf87a92b76924ecd50
Zhuo Su
admin_assigned
2023-12-07T11:56:35.247Z
655238b2957bf8a398eca2a1
/avatars/f0e5af3de113b998bcf84953ecd8b930.svg
Zhuo Su
user
hellozhuo
65713bdf87a92b76924ecd51
Yu Hong
65713bdf87a92b76924ecd52
Yingliang Zhang
65713bdf87a92b76924ecd53
Jingyi Yu
admin_assigned
2023-12-07T11:57:01.625Z
63ffcec6e7767a89533b8f80
/avatars/b9e8673943741c2e7b80b15a061bb951.svg
JINGYI Yu
user
Daluuu
65713bdf87a92b76924ecd54
Lan Xu
HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian
  Splatting
We have recently seen tremendous progress in photo-real human modeling and
rendering. Yet, efficiently rendering realistic human performance and
integrating it into the rasterization pipeline remains challenging. In this
paper, we present HiFi4G, an explicit and compact Gaussian-based approach for
high-fidelity human performance rendering from dense footage. Our core
intuition is to marry the 3D Gaussian representation with non-rigid tracking,
achieving a compact and compression-friendly representation. We first propose a
dual-graph mechanism to obtain motion priors, with a coarse deformation graph
for effective initialization and a fine-grained Gaussian graph to enforce
subsequent constraints. Then, we utilize a 4D Gaussian optimization scheme with
adaptive spatial-temporal regularizers to effectively balance the non-rigid
prior and Gaussian updating. We also present a companion compression scheme
with residual compensation for immersive experiences on various platforms. It
achieves a substantial compression rate of approximately 25 times, with less
than 2MB of storage per frame. Extensive experiments demonstrate the
effectiveness of our approach, which significantly outperforms existing
approaches in terms of optimization speed, rendering quality, and storage
overhead.
HiFi4G uses a Gaussian-based approach with 3D Gaussian representation and non-rigid tracking for efficient high-fidelity human performance rendering, offering significant compression and quality improvements.
Gaussian-based approach 3D Gaussian representation non-rigid tracking dual-graph mechanism coarse deformation graph fine-grained Gaussian graph 4D Gaussian optimization adaptive spatial-temporal regularizers residual compensation