TokenFlow: Consistent Diffusion Features for Consistent Video Editing
The generative AI revolution has recently expanded to videos. Nevertheless,
current state-of-the-art video models are still lagging behind image models in
terms of visual quality and user control over the generated content. In this
work, we present a framework that harnesses the power of a text-to-image
diffusion model for the task of text-driven video editing. Specifically, given
a source video and a target text-prompt, our method generates a high-quality
video that adheres to the target text, while preserving the spatial layout and
motion of the input video. Our method is based on a key observation that
consistency in the edited video can be obtained by enforcing consistency in the
diffusion feature space. We achieve this by explicitly propagating diffusion
features based on inter-frame correspondences, readily available in the model.
Thus, our framework does not require any training or fine-tuning, and can work
in conjunction with any off-the-shelf text-to-image editing method. We
demonstrate state-of-the-art editing results on a variety of real-world videos.
Webpage: https://diffusion-tokenflow.github.io/
64b9e3f6d4463c3d218bebe8
Michal Geyer
admin_assigned
2023-07-21T07:38:56.487Z
6304fc4a34c824b17250a347
/avatars/e2e2259a9cde8fd6981885892982f69e.svg
michal geyer
user
mishmish-g
64b9e3f6d4463c3d218bebe9
Omer Bar-Tal
admin_assigned
2023-07-21T07:39:12.500Z
62e29044a133a252b5cf70b2
/avatars/6d09ddcba9bc47c309150a8d77815891.svg
Omer Bar-Tal
user
omerbartal
64b9e3f6d4463c3d218bebea
Shai Bagon
64b9e3f6d4463c3d218bebeb
Tali Dekel
admin_assigned
2023-07-21T07:39:26.302Z
631cddec68f7da9ad24f6fc7
/avatars/7d4f1ce805e5889ca6594bd4a93f2583.svg
Tali Dekel
user
talidekel
TokenFlow: Consistent Diffusion Features for Consistent Video Editing
The generative AI revolution has recently expanded to videos. Nevertheless,
current state-of-the-art video models are still lagging behind image models in
terms of visual quality and user control over the generated content. In this
work, we present a framework that harnesses the power of a text-to-image
diffusion model for the task of text-driven video editing. Specifically, given
a source video and a target text-prompt, our method generates a high-quality
video that adheres to the target text, while preserving the spatial layout and
motion of the input video. Our method is based on a key observation that
consistency in the edited video can be obtained by enforcing consistency in the
diffusion feature space. We achieve this by explicitly propagating diffusion
features based on inter-frame correspondences, readily available in the model.
Thus, our framework does not require any training or fine-tuning, and can work
in conjunction with any off-the-shelf text-to-image editing method. We
demonstrate state-of-the-art editing results on a variety of real-world videos.
Webpage: https://diffusion-tokenflow.github.io/
A framework for text-driven video editing uses diffusion models to ensure consistency in the edited video by propagating features based on inter-frame correspondences.
text-to-image diffusion model text-driven video editing diffusion feature space inter-frame correspondences