Diffusion Priors for Dynamic View Synthesis from Monocular Videos
Dynamic novel view synthesis aims to capture the temporal evolution of visual
content within videos. Existing methods struggle to distinguishing between
motion and structure, particularly in scenarios where camera poses are either
unknown or constrained compared to object motion. Furthermore, with information
solely from reference images, it is extremely challenging to hallucinate unseen
regions that are occluded or partially observed in the given videos. To address
these issues, we first finetune a pretrained RGB-D diffusion model on the video
frames using a customization technique. Subsequently, we distill the knowledge
from the finetuned model to a 4D representations encompassing both dynamic and
static Neural Radiance Fields (NeRF) components. The proposed pipeline achieves
geometric consistency while preserving the scene identity. We perform thorough
experiments to evaluate the efficacy of the proposed method qualitatively and
quantitatively. Our results demonstrate the robustness and utility of our
approach in challenging cases, further advancing dynamic novel view synthesis.
65a0af0126d1e9df4fe416ce
Chaoyang Wang
admin_assigned
2024-01-12T10:16:02.817Z
64c166de11f3f1d23a99c6fd
/avatars/d4730f70a6f1cb51266862c7a8d54f77.svg
Chaoyang Wang
user
cwang9
65a0af0126d1e9df4fe416cf
Peiye Zhuang
admin_assigned
2024-01-12T10:16:08.804Z
63e473cf9db5da2dc1f4e9ff
/avatars/6973b4c959ef8bb72c40c611d384a78a.svg
peiye zhuang
user
Kelest
65a0af0126d1e9df4fe416d0
Aliaksandr Siarohin
admin_assigned
2024-01-12T10:16:15.192Z
64276311eb9a0ed86180715b
/avatars/76f933cd549f10e5e2db379de235d304.svg
Aliaksandr Siarohin
user
aliaksandr-siarohin
65a0af0126d1e9df4fe416d1
Junli Cao
admin_assigned
2024-01-12T10:16:21.963Z
63f54aa73aa49d8cb97b84bc
/avatars/c73c5870039611ab9162daad46a1ba20.svg
junli cao
user
jlcao2
65a0af0126d1e9df4fe416d2
Guocheng Qian
admin_assigned
2024-01-12T10:16:28.558Z
645fed74335c21d19f3bf76c
https://cdn-avatars.huggingface.co/v1/production/uploads/645fed74335c21d19f3bf76c/gwVsllRWtSHbg4a1erkdF.jpeg
Guocheng Qian
user
guochengqian
65a0af0126d1e9df4fe416d3
Hsin-Ying Lee
admin_assigned
2024-01-12T10:16:34.561Z
631bbd918de8e645af71eed9
/avatars/1cb94deadc288d07c571c9289085548c.svg
Hsin-Ying Lee
user
james371507
65a0af0126d1e9df4fe416d4
Sergey Tulyakov
Diffusion Priors for Dynamic View Synthesis from Monocular Videos
Dynamic novel view synthesis aims to capture the temporal evolution of visual
content within videos. Existing methods struggle to distinguishing between
motion and structure, particularly in scenarios where camera poses are either
unknown or constrained compared to object motion. Furthermore, with information
solely from reference images, it is extremely challenging to hallucinate unseen
regions that are occluded or partially observed in the given videos. To address
these issues, we first finetune a pretrained RGB-D diffusion model on the video
frames using a customization technique. Subsequently, we distill the knowledge
from the finetuned model to a 4D representations encompassing both dynamic and
static Neural Radiance Fields (NeRF) components. The proposed pipeline achieves
geometric consistency while preserving the scene identity. We perform thorough
experiments to evaluate the efficacy of the proposed method qualitatively and
quantitatively. Our results demonstrate the robustness and utility of our
approach in challenging cases, further advancing dynamic novel view synthesis.
A finetuned RGB-D diffusion model combined with dynamic and static Neural Radiance Fields (NeRF) enhances dynamic novel view synthesis by achieving geometric consistency and robust hallucination of unseen video regions.
RGB-D diffusion model Neural Radiance Fields NeRF dynamic novel view synthesis geometric consistency scene identity