VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing
Recently, diffusion-based generative models have achieved remarkable success
for image generation and edition. However, their use for video editing still
faces important limitations. This paper introduces VidEdit, a novel method for
zero-shot text-based video editing ensuring strong temporal and spatial
consistency. Firstly, we propose to combine atlas-based and pre-trained
text-to-image diffusion models to provide a training-free and efficient editing
method, which by design fulfills temporal smoothness. Secondly, we leverage
off-the-shelf panoptic segmenters along with edge detectors and adapt their use
for conditioned diffusion-based atlas editing. This ensures a fine spatial
control on targeted regions while strictly preserving the structure of the
original video. Quantitative and qualitative experiments show that VidEdit
outperforms state-of-the-art methods on DAVIS dataset, regarding semantic
faithfulness, image preservation, and temporal consistency metrics. With this
framework, processing a single video only takes approximately one minute, and
it can generate multiple compatible edits based on a unique text prompt.
Project web-page at https://videdit.github.io
648beeec10bced401c50e780
Paul Couairon
648beeec10bced401c50e781
Clément Rambour
648beeec10bced401c50e782
Jean-Emmanuel Haugeard
648beeec10bced401c50e783
Nicolas Thome
VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing
Recently, diffusion-based generative models have achieved remarkable success
for image generation and edition. However, their use for video editing still
faces important limitations. This paper introduces VidEdit, a novel method for
zero-shot text-based video editing ensuring strong temporal and spatial
consistency. Firstly, we propose to combine atlas-based and pre-trained
text-to-image diffusion models to provide a training-free and efficient editing
method, which by design fulfills temporal smoothness. Secondly, we leverage
off-the-shelf panoptic segmenters along with edge detectors and adapt their use
for conditioned diffusion-based atlas editing. This ensures a fine spatial
control on targeted regions while strictly preserving the structure of the
original video. Quantitative and qualitative experiments show that VidEdit
outperforms state-of-the-art methods on DAVIS dataset, regarding semantic
faithfulness, image preservation, and temporal consistency metrics. With this
framework, processing a single video only takes approximately one minute, and
it can generate multiple compatible edits based on a unique text prompt.
Project web-page at https://videdit.github.io
diffusion-based generative models zero-shot text-based video editing atlas-based text-to-image diffusion models training-free temporal smoothness off-the-shelf panoptic segmenters edge detectors conditioned diffusion-based atlas editing semantic faithfulness image preservation temporal consistency metrics