V^3: Viewing Volumetric Videos on Mobiles via Streamable 2D Dynamic
  Gaussians
Experiencing high-fidelity volumetric video as seamlessly as 2D videos is a
long-held dream. However, current dynamic 3DGS methods, despite their high
rendering quality, face challenges in streaming on mobile devices due to
computational and bandwidth constraints. In this paper, we introduce
V3(Viewing Volumetric Videos), a novel approach that enables
high-quality mobile rendering through the streaming of dynamic Gaussians. Our
key innovation is to view dynamic 3DGS as 2D videos, facilitating the use of
hardware video codecs. Additionally, we propose a two-stage training strategy
to reduce storage requirements with rapid training speed. The first stage
employs hash encoding and shallow MLP to learn motion, then reduces the number
of Gaussians through pruning to meet the streaming requirements, while the
second stage fine tunes other Gaussian attributes using residual entropy loss
and temporal loss to improve temporal continuity. This strategy, which
disentangles motion and appearance, maintains high rendering quality with
compact storage requirements. Meanwhile, we designed a multi-platform player to
decode and render 2D Gaussian videos. Extensive experiments demonstrate the
effectiveness of V3, outperforming other methods by enabling
high-quality rendering and streaming on common devices, which is unseen before.
As the first to stream dynamic Gaussians on mobile devices, our companion
player offers users an unprecedented volumetric video experience, including
smooth scrolling and instant sharing. Our project page with source code is
available at https://authoritywang.github.io/v3/.
66f0dfa1507a27eac7ce7acd
Penghao Wang
claimed_verified
2024-10-21T07:49:59.490Z
6462120032083c28b408e754
/avatars/f02c6d23f03420b716d995c92f8e2213.svg
Penghao Wang
user
AuWang
66f0dfa1507a27eac7ce7ace
Zhirui Zhang
66f0dfa1507a27eac7ce7acf
Liao Wang
admin_assigned
2024-09-23T07:48:37.407Z
64180cb3db24526c7c9b011e
/avatars/bcbfe1e18a2c6402d5db14520279b04b.svg
wliao
user
liaowang11
66f0dfa1507a27eac7ce7ad0
Kaixin Yao
66f0dfa1507a27eac7ce7ad1
Siyuan Xie
66f0dfa1507a27eac7ce7ad2
Jingyi Yu
admin_assigned
2024-09-23T07:49:12.361Z
63ffcec6e7767a89533b8f80
/avatars/b9e8673943741c2e7b80b15a061bb951.svg
JINGYI Yu
user
Daluuu
66f0dfa1507a27eac7ce7ad3
Minye Wu
admin_assigned
2024-09-23T07:49:18.290Z
656ef1678d218bc3ce4b7845
/avatars/09e4091f486024c5fd97f77339096f8f.svg
Minye Wu
user
wuminye
66f0dfa1507a27eac7ce7ad4
Lan Xu
V^3: Viewing Volumetric Videos on Mobiles via Streamable 2D Dynamic
  Gaussians
Experiencing high-fidelity volumetric video as seamlessly as 2D videos is a
long-held dream. However, current dynamic 3DGS methods, despite their high
rendering quality, face challenges in streaming on mobile devices due to
computational and bandwidth constraints. In this paper, we introduce
V3(Viewing Volumetric Videos), a novel approach that enables
high-quality mobile rendering through the streaming of dynamic Gaussians. Our
key innovation is to view dynamic 3DGS as 2D videos, facilitating the use of
hardware video codecs. Additionally, we propose a two-stage training strategy
to reduce storage requirements with rapid training speed. The first stage
employs hash encoding and shallow MLP to learn motion, then reduces the number
of Gaussians through pruning to meet the streaming requirements, while the
second stage fine tunes other Gaussian attributes using residual entropy loss
and temporal loss to improve temporal continuity. This strategy, which
disentangles motion and appearance, maintains high rendering quality with
compact storage requirements. Meanwhile, we designed a multi-platform player to
decode and render 2D Gaussian videos. Extensive experiments demonstrate the
effectiveness of V3, outperforming other methods by enabling
high-quality rendering and streaming on common devices, which is unseen before.
As the first to stream dynamic Gaussians on mobile devices, our companion
player offers users an unprecedented volumetric video experience, including
smooth scrolling and instant sharing. Our project page with source code is
available at https://authoritywang.github.io/v3/.
A novel approach, V³, enables high-quality mobile rendering and streaming of volumetric videos by streaming dynamic Gaussians as 2D videos, using hardware video codecs and a two-stage training strategy to optimize motion and appearance.
dynamic 3DGS volumetric video mobile rendering streaming computational constraints bandwidth constraints hash encoding shallow MLP pruning residual entropy loss temporal loss multi-platform player decoding rendering smooth scrolling instant sharing