Animate3D: Animating Any 3D Model with Multi-view Video Diffusion
Recent advances in 4D generation mainly focus on generating 4D content by
distilling pre-trained text or single-view image-conditioned models. It is
inconvenient for them to take advantage of various off-the-shelf 3D assets with
multi-view attributes, and their results suffer from spatiotemporal
inconsistency owing to the inherent ambiguity in the supervision signals. In
this work, we present Animate3D, a novel framework for animating any static 3D
model. The core idea is two-fold: 1) We propose a novel multi-view video
diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D
object, which is trained on our presented large-scale multi-view video dataset
(MV-Video). 2) Based on MV-VDM, we introduce a framework combining
reconstruction and 4D Score Distillation Sampling (4D-SDS) to leverage the
multi-view video diffusion priors for animating 3D objects. Specifically, for
MV-VDM, we design a new spatiotemporal attention module to enhance spatial and
temporal consistency by integrating 3D and video diffusion models.
Additionally, we leverage the static 3D model's multi-view renderings as
conditions to preserve its identity. For animating 3D models, an effective
two-stage pipeline is proposed: we first reconstruct motions directly from
generated multi-view videos, followed by the introduced 4D-SDS to refine both
appearance and motion. Qualitative and quantitative experiments demonstrate
that Animate3D significantly outperforms previous approaches. Data, code, and
models will be open-released.
669739888ee837fcda28bac8
Yanqin Jiang
claimed_verified
2024-10-10T08:02:30.374Z
63738b5d3e96500368852771
/avatars/0bf8464652751a5e355ee2a86498171f.svg
yanqinJiang
user
yanqinJiang
669739888ee837fcda28bac9
Chaohui Yu
669739888ee837fcda28baca
Chenjie Cao
669739888ee837fcda28bacb
Fan Wang
669739888ee837fcda28bacc
Weiming Hu
669739888ee837fcda28bacd
Jin Gao
Animate3D: Animating Any 3D Model with Multi-view Video Diffusion
Recent advances in 4D generation mainly focus on generating 4D content by
distilling pre-trained text or single-view image-conditioned models. It is
inconvenient for them to take advantage of various off-the-shelf 3D assets with
multi-view attributes, and their results suffer from spatiotemporal
inconsistency owing to the inherent ambiguity in the supervision signals. In
this work, we present Animate3D, a novel framework for animating any static 3D
model. The core idea is two-fold: 1) We propose a novel multi-view video
diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D
object, which is trained on our presented large-scale multi-view video dataset
(MV-Video). 2) Based on MV-VDM, we introduce a framework combining
reconstruction and 4D Score Distillation Sampling (4D-SDS) to leverage the
multi-view video diffusion priors for animating 3D objects. Specifically, for
MV-VDM, we design a new spatiotemporal attention module to enhance spatial and
temporal consistency by integrating 3D and video diffusion models.
Additionally, we leverage the static 3D model's multi-view renderings as
conditions to preserve its identity. For animating 3D models, an effective
two-stage pipeline is proposed: we first reconstruct motions directly from
generated multi-view videos, followed by the introduced 4D-SDS to refine both
appearance and motion. Qualitative and quantitative experiments demonstrate
that Animate3D significantly outperforms previous approaches. Data, code, and
models will be open-released.
Animate3D uses a novel multi-view video diffusion model trained on multi-view videos to enhance the animation of static 3D models, improving spatiotemporal consistency and appearance fidelity.
multi-view video diffusion model MV-VDM large-scale multi-view video dataset MV-Video spatiotemporal attention module 4D Score Distillation Sampling 4D-SDS multi-view renderings two-stage pipeline