Gaussian Variation Field Diffusion for High-fidelity Video-to-4D
  Synthesis
In this paper, we present a novel framework for video-to-4D generation that
creates high-quality dynamic 3D content from single video inputs. Direct 4D
diffusion modeling is extremely challenging due to costly data construction and
the high-dimensional nature of jointly representing 3D shape, appearance, and
motion. We address these challenges by introducing a Direct 4DMesh-to-GS
Variation Field VAE that directly encodes canonical Gaussian Splats (GS) and
their temporal variations from 3D animation data without per-instance fitting,
and compresses high-dimensional animations into a compact latent space.
Building upon this efficient representation, we train a Gaussian Variation
Field diffusion model with temporal-aware Diffusion Transformer conditioned on
input videos and canonical GS. Trained on carefully-curated animatable 3D
objects from the Objaverse dataset, our model demonstrates superior generation
quality compared to existing methods. It also exhibits remarkable
generalization to in-the-wild video inputs despite being trained exclusively on
synthetic data, paving the way for generating high-quality animated 3D content.
Project page: https://gvfdiffusion.github.io/.
6892cf048da45ffb0a2b24b6
Bowen Zhang
claimed_verified
2025-08-06T19:16:22.835Z
6237f25f16004228e6c74e01
/avatars/cc63ce464a25702c8155610d2a708595.svg
Bowen Zhang
user
BwZhang
6892cf048da45ffb0a2b24b7
Sicheng Xu
6892cf048da45ffb0a2b24b8
Chuxin Wang
6892cf048da45ffb0a2b24b9
Jiaolong Yang
6892cf048da45ffb0a2b24ba
Feng Zhao
6892cf048da45ffb0a2b24bb
Dong Chen
6892cf048da45ffb0a2b24bc
Baining Guo
Gaussian Variation Field Diffusion for High-fidelity Video-to-4D
  Synthesis
In this paper, we present a novel framework for video-to-4D generation that
creates high-quality dynamic 3D content from single video inputs. Direct 4D
diffusion modeling is extremely challenging due to costly data construction and
the high-dimensional nature of jointly representing 3D shape, appearance, and
motion. We address these challenges by introducing a Direct 4DMesh-to-GS
Variation Field VAE that directly encodes canonical Gaussian Splats (GS) and
their temporal variations from 3D animation data without per-instance fitting,
and compresses high-dimensional animations into a compact latent space.
Building upon this efficient representation, we train a Gaussian Variation
Field diffusion model with temporal-aware Diffusion Transformer conditioned on
input videos and canonical GS. Trained on carefully-curated animatable 3D
objects from the Objaverse dataset, our model demonstrates superior generation
quality compared to existing methods. It also exhibits remarkable
generalization to in-the-wild video inputs despite being trained exclusively on
synthetic data, paving the way for generating high-quality animated 3D content.
Project page: https://gvfdiffusion.github.io/.
A novel framework uses a Direct 4DMesh-to-GS Variation Field VAE and Gaussian Variation Field diffusion model to generate high-quality dynamic 3D content from single video inputs, demonstrating superior quality and generalization.
Direct 4D diffusion modeling Gaussian Splats (GS) Direct 4DMesh-to-GS Variation Field VAE Gaussian Variation Field diffusion model Diffusion Transformer Objaverse dataset