PLA4D: Pixel-Level Alignments for Text-to-4D Gaussian Splatting
As text-conditioned diffusion models (DMs) achieve breakthroughs in image,
video, and 3D generation, the research community's focus has shifted to the
more challenging task of text-to-4D synthesis, which introduces a temporal
dimension to generate dynamic 3D objects. In this context, we identify Score
Distillation Sampling (SDS), a widely used technique for text-to-3D synthesis,
as a significant hindrance to text-to-4D performance due to its Janus-faced and
texture-unrealistic problems coupled with high computational costs. In this
paper, we propose Pixel-Level Alignments for
Text-to-4D Gaussian Splatting (PLA4D), a novel method that
utilizes text-to-video frames as explicit pixel alignment targets to generate
static 3D objects and inject motion into them. Specifically, we introduce Focal
Alignment to calibrate camera poses for rendering and GS-Mesh Contrastive
Learning to distill geometry priors from rendered image contrasts at the pixel
level. Additionally, we develop Motion Alignment using a deformation network to
drive changes in Gaussians and implement Reference Refinement for smooth 4D
object surfaces. These techniques enable 4D Gaussian Splatting to align
geometry, texture, and motion with generated videos at the pixel level.
Compared to previous methods, PLA4D produces synthesized outputs with better
texture details in less time and effectively mitigates the Janus-faced problem.
PLA4D is fully implemented using open-source models, offering an accessible,
user-friendly, and promising direction for 4D digital content creation. Our
project page:
https://github.com/MiaoQiaowei/PLA4D.github.io{https://github.com/MiaoQiaowei/PLA4D.github.io}.
66594d28b8c81d7357289946
Qiaowei Miao
admin_assigned
2024-05-31T10:06:27.770Z
63dcb90622cc06e76a8341eb
/avatars/59a82bae9792df53af098972285ec19a.svg
Qiaowei Miao
user
wdmmqw
66594d28b8c81d7357289947
Yawei Luo
claimed_verified
2024-06-02T18:47:25.711Z
665bfc4c88912c5ab646c3c9
/avatars/db284af3be943b5786cf0026279ef3b9.svg
Yawei Luo
user
RoyalVane
66594d28b8c81d7357289948
Yi Yang
PLA4D: Pixel-Level Alignments for Text-to-4D Gaussian Splatting
As text-conditioned diffusion models (DMs) achieve breakthroughs in image,
video, and 3D generation, the research community's focus has shifted to the
more challenging task of text-to-4D synthesis, which introduces a temporal
dimension to generate dynamic 3D objects. In this context, we identify Score
Distillation Sampling (SDS), a widely used technique for text-to-3D synthesis,
as a significant hindrance to text-to-4D performance due to its Janus-faced and
texture-unrealistic problems coupled with high computational costs. In this
paper, we propose Pixel-Level Alignments for
Text-to-4D Gaussian Splatting (PLA4D), a novel method that
utilizes text-to-video frames as explicit pixel alignment targets to generate
static 3D objects and inject motion into them. Specifically, we introduce Focal
Alignment to calibrate camera poses for rendering and GS-Mesh Contrastive
Learning to distill geometry priors from rendered image contrasts at the pixel
level. Additionally, we develop Motion Alignment using a deformation network to
drive changes in Gaussians and implement Reference Refinement for smooth 4D
object surfaces. These techniques enable 4D Gaussian Splatting to align
geometry, texture, and motion with generated videos at the pixel level.
Compared to previous methods, PLA4D produces synthesized outputs with better
texture details in less time and effectively mitigates the Janus-faced problem.
PLA4D is fully implemented using open-source models, offering an accessible,
user-friendly, and promising direction for 4D digital content creation. Our
project page:
https://github.com/MiaoQiaowei/PLA4D.github.io{https://github.com/MiaoQiaowei/PLA4D.github.io}.
PLA4D advances text-to-4D synthesis by improving texture, geometry, and motion alignment through novel pixel-level techniques.
diffusion models text-to-4D synthesis Score Distillation Sampling Pixel-Level Alignments for Text-to-4D Gaussian Splatting Focal Alignment GS-Mesh Contrastive Learning Motion Alignment deformation network Reference Refinement 4D Gaussian Splatting text-to-video frames