Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed
  Diffusion Models
Text-guided diffusion models have revolutionized image and video generation
and have also been successfully used for optimization-based 3D object
synthesis. Here, we instead focus on the underexplored text-to-4D setting and
synthesize dynamic, animated 3D objects using score distillation methods with
an additional temporal dimension. Compared to previous work, we pursue a novel
compositional generation-based approach, and combine text-to-image,
text-to-video, and 3D-aware multiview diffusion models to provide feedback
during 4D object optimization, thereby simultaneously enforcing temporal
consistency, high-quality visual appearance and realistic geometry. Our method,
called Align Your Gaussians (AYG), leverages dynamic 3D Gaussian Splatting with
deformation fields as 4D representation. Crucial to AYG is a novel method to
regularize the distribution of the moving 3D Gaussians and thereby stabilize
the optimization and induce motion. We also propose a motion amplification
mechanism as well as a new autoregressive synthesis scheme to generate and
combine multiple 4D sequences for longer generation. These techniques allow us
to synthesize vivid dynamic scenes, outperform previous work qualitatively and
quantitatively and achieve state-of-the-art text-to-4D performance. Due to the
Gaussian 4D representation, different 4D animations can be seamlessly combined,
as we demonstrate. AYG opens up promising avenues for animation, simulation and
digital content creation as well as synthetic data generation.
6584e8d7ebd17631e9921c6e
Huan Ling
admin_assigned
2023-12-22T12:02:21.125Z
6344bf36762379fc6302abed
/avatars/f4442fef316ac96652d4f99238822209.svg
huanling
user
huanling
6584e8d7ebd17631e9921c6f
Seung Wook Kim
admin_assigned
2023-12-22T12:02:28.075Z
64fb5827d82fc6977d70582d
/avatars/54180874d2f162572c414970b8750759.svg
Seung Wook Kim
user
swkimbd
6584e8d7ebd17631e9921c70
Antonio Torralba
6584e8d7ebd17631e9921c71
Sanja Fidler
6584e8d7ebd17631e9921c72
Karsten Kreis
Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed
  Diffusion Models
Text-guided diffusion models have revolutionized image and video generation
and have also been successfully used for optimization-based 3D object
synthesis. Here, we instead focus on the underexplored text-to-4D setting and
synthesize dynamic, animated 3D objects using score distillation methods with
an additional temporal dimension. Compared to previous work, we pursue a novel
compositional generation-based approach, and combine text-to-image,
text-to-video, and 3D-aware multiview diffusion models to provide feedback
during 4D object optimization, thereby simultaneously enforcing temporal
consistency, high-quality visual appearance and realistic geometry. Our method,
called Align Your Gaussians (AYG), leverages dynamic 3D Gaussian Splatting with
deformation fields as 4D representation. Crucial to AYG is a novel method to
regularize the distribution of the moving 3D Gaussians and thereby stabilize
the optimization and induce motion. We also propose a motion amplification
mechanism as well as a new autoregressive synthesis scheme to generate and
combine multiple 4D sequences for longer generation. These techniques allow us
to synthesize vivid dynamic scenes, outperform previous work qualitatively and
quantitatively and achieve state-of-the-art text-to-4D performance. Due to the
Gaussian 4D representation, different 4D animations can be seamlessly combined,
as we demonstrate. AYG opens up promising avenues for animation, simulation and
digital content creation as well as synthetic data generation.
A text-guided compositional generation method uses score distillation and dynamic 3D Gaussian Splatting to synthesize high-quality, animated 4D objects with improved temporal consistency and realism.
text-guided diffusion models score distillation text-to-4D compositional generation text-to-image text-to-video 3D-aware multiview diffusion models 4D object optimization temporal consistency visual appearance realistic geometry Align Your Gaussians dynamic 3D Gaussian Splatting deformation fields Gaussian 4D representation motion amplification autoregressive synthesis 4D sequences synthetic data generation