GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation
Creating 4D fields of Gaussian Splatting from images or videos is a
challenging task due to its under-constrained nature. While the optimization
can draw photometric reference from the input videos or be regulated by
generative models, directly supervising Gaussian motions remains underexplored.
In this paper, we introduce a novel concept, Gaussian flow, which connects the
dynamics of 3D Gaussians and pixel velocities between consecutive frames. The
Gaussian flow can be efficiently obtained by splatting Gaussian dynamics into
the image space. This differentiable process enables direct dynamic supervision
from optical flow. Our method significantly benefits 4D dynamic content
generation and 4D novel view synthesis with Gaussian Splatting, especially for
contents with rich motions that are hard to be handled by existing methods. The
common color drifting issue that happens in 4D generation is also resolved with
improved Guassian dynamics. Superior visual quality on extensive experiments
demonstrates our method's effectiveness. Quantitative and qualitative
evaluations show that our method achieves state-of-the-art results on both
tasks of 4D generation and 4D novel view synthesis. Project page:
https://zerg-overmind.github.io/GaussianFlow.github.io/
65fa6512af74a079c161ecab
Quankai Gao
extracted_confirmed
2024-03-20T04:25:42.331Z
64c12bb1f149752445c39472
/avatars/8a534f5b850065f72dbe2e036181f7a9.svg
Quankai Gao
user
Quankai
65fa6512af74a079c161ecac
Qiangeng Xu
claimed_verified
2024-03-21T08:27:10.225Z
64c1a214c84cd12dd79c1530
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/CSK0AyQhXs3Izrnt9s6_l.png
qiangeng xu
user
xharlie
65fa6512af74a079c161ecad
Zhe Cao
admin_assigned
2024-03-21T12:52:42.590Z
6526b4486e00e9a9d6552c9a
/avatars/d00d93611610432ceee7b976155e013e.svg
Zhe Cao
user
huskyc
65fa6512af74a079c161ecae
Ben Mildenhall
65fa6512af74a079c161ecaf
Wenchao Ma
65fa6512af74a079c161ecb0
Le Chen
65fa6512af74a079c161ecb1
Danhang Tang
65fa6512af74a079c161ecb2
Ulrich Neumann
GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation
Creating 4D fields of Gaussian Splatting from images or videos is a
challenging task due to its under-constrained nature. While the optimization
can draw photometric reference from the input videos or be regulated by
generative models, directly supervising Gaussian motions remains underexplored.
In this paper, we introduce a novel concept, Gaussian flow, which connects the
dynamics of 3D Gaussians and pixel velocities between consecutive frames. The
Gaussian flow can be efficiently obtained by splatting Gaussian dynamics into
the image space. This differentiable process enables direct dynamic supervision
from optical flow. Our method significantly benefits 4D dynamic content
generation and 4D novel view synthesis with Gaussian Splatting, especially for
contents with rich motions that are hard to be handled by existing methods. The
common color drifting issue that happens in 4D generation is also resolved with
improved Guassian dynamics. Superior visual quality on extensive experiments
demonstrates our method's effectiveness. Quantitative and qualitative
evaluations show that our method achieves state-of-the-art results on both
tasks of 4D generation and 4D novel view synthesis. Project page:
https://zerg-overmind.github.io/GaussianFlow.github.io/
Gaussian flow connects 3D Gaussian dynamics and pixel velocities, enabling effective 4D content generation and novel view synthesis with improved Gaussian dynamics.
Gaussian Splatting Gaussian flow 3D Gaussians pixel velocities optical flow 4D dynamic content generation 4D novel view synthesis color drifting Gaussian dynamics