Hierarchical Masked 3D Diffusion Model for Video Outpainting
Video outpainting aims to adequately complete missing areas at the edges of
video frames. Compared to image outpainting, it presents an additional
challenge as the model should maintain the temporal consistency of the filled
area. In this paper, we introduce a masked 3D diffusion model for video
outpainting. We use the technique of mask modeling to train the 3D diffusion
model. This allows us to use multiple guide frames to connect the results of
multiple video clip inferences, thus ensuring temporal consistency and reducing
jitter between adjacent frames. Meanwhile, we extract the global frames of the
video as prompts and guide the model to obtain information other than the
current video clip using cross-attention. We also introduce a hybrid
coarse-to-fine inference pipeline to alleviate the artifact accumulation
problem. The existing coarse-to-fine pipeline only uses the infilling strategy,
which brings degradation because the time interval of the sparse frames is too
large. Our pipeline benefits from bidirectional learning of the mask modeling
and thus can employ a hybrid strategy of infilling and interpolation when
generating sparse frames. Experiments show that our method achieves
state-of-the-art results in video outpainting tasks. More results are provided
at our https://fanfanda.github.io/M3DDM/.
64f830998a4c5c401d7d8aef
Fanda Fan
64f830998a4c5c401d7d8af0
Chaoxu Guo
64f830998a4c5c401d7d8af1
Litong Gong
64f830998a4c5c401d7d8af2
Biao Wang
claimed_verified
2024-03-06T15:35:27.672Z
64dcd9de0a0efd28e3b846e3
/avatars/0732be326b1d3658118f669dd283d582.svg
ictwangbiao
user
ictwangbiao
64f830998a4c5c401d7d8af3
Tiezheng Ge
64f830998a4c5c401d7d8af4
Yuning Jiang
64f830998a4c5c401d7d8af5
Chunjie Luo
64f830998a4c5c401d7d8af6
Jianfeng Zhan
Hierarchical Masked 3D Diffusion Model for Video Outpainting
Video outpainting aims to adequately complete missing areas at the edges of
video frames. Compared to image outpainting, it presents an additional
challenge as the model should maintain the temporal consistency of the filled
area. In this paper, we introduce a masked 3D diffusion model for video
outpainting. We use the technique of mask modeling to train the 3D diffusion
model. This allows us to use multiple guide frames to connect the results of
multiple video clip inferences, thus ensuring temporal consistency and reducing
jitter between adjacent frames. Meanwhile, we extract the global frames of the
video as prompts and guide the model to obtain information other than the
current video clip using cross-attention. We also introduce a hybrid
coarse-to-fine inference pipeline to alleviate the artifact accumulation
problem. The existing coarse-to-fine pipeline only uses the infilling strategy,
which brings degradation because the time interval of the sparse frames is too
large. Our pipeline benefits from bidirectional learning of the mask modeling
and thus can employ a hybrid strategy of infilling and interpolation when
generating sparse frames. Experiments show that our method achieves
state-of-the-art results in video outpainting tasks. More results are provided
at our https://fanfanda.github.io/M3DDM/.
A masked 3D diffusion model with cross-attention and a hybrid coarse-to-fine pipeline achieves state-of-the-art results in video outpainting by maintaining temporal consistency and reducing artifacts.
diffusion model mask modeling temporal consistency cross-attention coarse-to-fine pipeline infilling interpolation Masked 3D diffusion model M3DDM