MaskINT: Video Editing via Interpolative Non-autoregressive Masked
  Transformers
Recent advances in generative AI have significantly enhanced image and video
editing, particularly in the context of text prompt control. State-of-the-art
approaches predominantly rely on diffusion models to accomplish these tasks.
However, the computational demands of diffusion-based methods are substantial,
often necessitating large-scale paired datasets for training, and therefore
challenging the deployment in practical applications. This study addresses this
challenge by breaking down the text-based video editing process into two
separate stages. In the first stage, we leverage an existing text-to-image
diffusion model to simultaneously edit a few keyframes without additional
fine-tuning. In the second stage, we introduce an efficient model called
MaskINT, which is built on non-autoregressive masked generative transformers
and specializes in frame interpolation between the keyframes, benefiting from
structural guidance provided by intermediate frames. Our comprehensive set of
experiments illustrates the efficacy and efficiency of MaskINT when compared to
other diffusion-based methodologies. This research offers a practical solution
for text-based video editing and showcases the potential of non-autoregressive
masked generative transformers in this domain.
6583c1acc73f74776b03d2f5
Haoyu Ma
admin_assigned
2023-12-21T09:36:23.502Z
650a8979c19e5b4c8a6ff062
https://cdn-avatars.huggingface.co/v1/production/uploads/650a8979c19e5b4c8a6ff062/64_JuECX_k_-uK7m7nlua.jpeg
Haoyu Ma
user
haoyum1997
6583c1acc73f74776b03d2f6
Shahin Mahdizadehaghdam
claimed_verified
2023-12-22T07:36:18.457Z
6584b71cb0b0e7e38a1e1b7c
/avatars/52080f076ddaba4cb11c3779f1e8654f.svg
Shahin Mahdizadehaghdam
user
shahinma
6583c1acc73f74776b03d2f7
Bichen Wu
admin_assigned
2023-12-21T09:36:48.563Z
6363749520c2cf37bccd9e84
/avatars/e9de7911e67090d883f15a7a58f18e4e.svg
Bichen Wu
user
bichenwu
6583c1acc73f74776b03d2f8
Zhipeng Fan
6583c1acc73f74776b03d2f9
Yuchao Gu
admin_assigned
2023-12-21T09:37:38.956Z
63021630a35b21bd8a53305a
/avatars/7a7e8b39749eda61e57d8a1908726558.svg
Gu Yuchao
user
guyuchao
6583c1acc73f74776b03d2fa
Wenliang Zhao
admin_assigned
2023-12-21T09:37:48.372Z
6306e962660f01f150a90776
/avatars/095017b35a5ab0933bbd802d18e96fac.svg
Wenliang Zhao
user
wl-zhao
6583c1acc73f74776b03d2fb
Lior Shapira
admin_assigned
2023-12-21T09:37:56.098Z
6350666a0f376d3c482af8e6
/avatars/bf71d055c669de89a5ad4d6cba6280c0.svg
Lior Shapira
user
liorshap77
6583c1acc73f74776b03d2fc
Xiaohui Xie
admin_assigned
2023-12-21T09:38:03.348Z
64220275ab125f65e85e4c49
/avatars/fae99aa1abca4d2eb170100bbe462d6e.svg
Xiaohui Xie
user
xhx
MaskINT: Video Editing via Interpolative Non-autoregressive Masked
  Transformers
Recent advances in generative AI have significantly enhanced image and video
editing, particularly in the context of text prompt control. State-of-the-art
approaches predominantly rely on diffusion models to accomplish these tasks.
However, the computational demands of diffusion-based methods are substantial,
often necessitating large-scale paired datasets for training, and therefore
challenging the deployment in practical applications. This study addresses this
challenge by breaking down the text-based video editing process into two
separate stages. In the first stage, we leverage an existing text-to-image
diffusion model to simultaneously edit a few keyframes without additional
fine-tuning. In the second stage, we introduce an efficient model called
MaskINT, which is built on non-autoregressive masked generative transformers
and specializes in frame interpolation between the keyframes, benefiting from
structural guidance provided by intermediate frames. Our comprehensive set of
experiments illustrates the efficacy and efficiency of MaskINT when compared to
other diffusion-based methodologies. This research offers a practical solution
for text-based video editing and showcases the potential of non-autoregressive
masked generative transformers in this domain.
The study introduces MaskINT, a non-autoregressive masked generative transformer, to efficiently edit videos based on text prompts by combining it with an existing text-to-image diffusion model for keyframe editing.
diffusion models text-to-image MaskINT non-autoregressive masked generative transformers frame interpolation keyframes