4Diffusion: Multi-view Video Diffusion Model for 4D Generation
Current 4D generation methods have achieved noteworthy efficacy with the aid
of advanced diffusion generative models. However, these methods lack multi-view
spatial-temporal modeling and encounter challenges in integrating diverse prior
knowledge from multiple diffusion models, resulting in inconsistent temporal
appearance and flickers. In this paper, we propose a novel 4D generation
pipeline, namely 4Diffusion aimed at generating spatial-temporally consistent
4D content from a monocular video. We first design a unified diffusion model
tailored for multi-view video generation by incorporating a learnable motion
module into a frozen 3D-aware diffusion model to capture multi-view
spatial-temporal correlations. After training on a curated dataset, our
diffusion model acquires reasonable temporal consistency and inherently
preserves the generalizability and spatial consistency of the 3D-aware
diffusion model. Subsequently, we propose 4D-aware Score Distillation Sampling
loss, which is based on our multi-view video diffusion model, to optimize 4D
representation parameterized by dynamic NeRF. This aims to eliminate
discrepancies arising from multiple diffusion models, allowing for generating
spatial-temporally consistent 4D content. Moreover, we devise an anchor loss to
enhance the appearance details and facilitate the learning of dynamic NeRF.
Extensive qualitative and quantitative experiments demonstrate that our method
achieves superior performance compared to previous methods.
665d347a6c20caf0909ccbc9
Haiyu Zhang
claimed_verified
2024-06-03T09:51:12.836Z
645b8bf6438d6cfbe1ae47ae
/avatars/3e4dc0523ba8e314b298567cf8f0c0ae.svg
Haiyu Zhang
user
aejion
665d347a6c20caf0909ccbca
Xinyuan Chen
admin_assigned
2024-06-03T09:04:38.163Z
643e943a70c6a27621eb1c89
/avatars/73ec521ab5ba84cc7908c52c0acef6ef.svg
Xinyuan Chen
user
AriaChen
665d347a6c20caf0909ccbcb
Yaohui Wang
admin_assigned
2024-06-03T09:05:01.858Z
63201256c6b20f03c829c4b8
/avatars/a42092119777d65e60b12eb5ba0e45f1.svg
Yaohui Wang
user
YaohuiW
665d347a6c20caf0909ccbcc
Xihui Liu
admin_assigned
2024-06-03T09:05:09.127Z
65d5ec74cd05bc1eaa125040
/avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
Xihui Liu
user
XihuiLiu
665d347a6c20caf0909ccbcd
Yunhong Wang
665d347a6c20caf0909ccbce
Yu Qiao
4Diffusion: Multi-view Video Diffusion Model for 4D Generation
Current 4D generation methods have achieved noteworthy efficacy with the aid
of advanced diffusion generative models. However, these methods lack multi-view
spatial-temporal modeling and encounter challenges in integrating diverse prior
knowledge from multiple diffusion models, resulting in inconsistent temporal
appearance and flickers. In this paper, we propose a novel 4D generation
pipeline, namely 4Diffusion aimed at generating spatial-temporally consistent
4D content from a monocular video. We first design a unified diffusion model
tailored for multi-view video generation by incorporating a learnable motion
module into a frozen 3D-aware diffusion model to capture multi-view
spatial-temporal correlations. After training on a curated dataset, our
diffusion model acquires reasonable temporal consistency and inherently
preserves the generalizability and spatial consistency of the 3D-aware
diffusion model. Subsequently, we propose 4D-aware Score Distillation Sampling
loss, which is based on our multi-view video diffusion model, to optimize 4D
representation parameterized by dynamic NeRF. This aims to eliminate
discrepancies arising from multiple diffusion models, allowing for generating
spatial-temporally consistent 4D content. Moreover, we devise an anchor loss to
enhance the appearance details and facilitate the learning of dynamic NeRF.
Extensive qualitative and quantitative experiments demonstrate that our method
achieves superior performance compared to previous methods.
A novel 4D generation pipeline, 4Diffusion, uses a unified diffusion model with a learnable motion module to generate spatial-temporally consistent 4D content from monocular video, and introduces a 4D-aware Score Distillation Sampling loss and anchor loss to improve performance.
diffusion generative models unified diffusion model multi-view spatial-temporal correlations frozen 3D-aware diffusion model 4D-aware Score Distillation Sampling loss dynamic NeRF anchor loss