AnyV2V: A Plug-and-Play Framework For Any Video-to-Video Editing Tasks
Video-to-video editing involves editing a source video along with additional
control (such as text prompts, subjects, or styles) to generate a new video
that aligns with the source video and the provided control. Traditional methods
have been constrained to certain editing types, limiting their ability to meet
the wide range of user demands. In this paper, we introduce AnyV2V, a novel
training-free framework designed to simplify video editing into two primary
steps: (1) employing an off-the-shelf image editing model (e.g.
InstructPix2Pix, InstantID, etc) to modify the first frame, (2) utilizing an
existing image-to-video generation model (e.g. I2VGen-XL) for DDIM inversion
and feature injection. In the first stage, AnyV2V can plug in any existing
image editing tools to support an extensive array of video editing tasks.
Beyond the traditional prompt-based editing methods, AnyV2V also can support
novel video editing tasks, including reference-based style transfer,
subject-driven editing, and identity manipulation, which were unattainable by
previous methods. In the second stage, AnyV2V can plug in any existing
image-to-video models to perform DDIM inversion and intermediate feature
injection to maintain the appearance and motion consistency with the source
video. On the prompt-based editing, we show that AnyV2V can outperform the
previous best approach by 35\% on prompt alignment, and 25\% on human
preference. On the three novel tasks, we show that AnyV2V also achieves a high
success rate. We believe AnyV2V will continue to thrive due to its ability to
seamlessly integrate the fast-evolving image editing methods. Such
compatibility can help AnyV2V to increase its versatility to cater to diverse
user demands.
65fcf07da2d58888315a2e97
Max Ku
claimed_verified
2024-03-25T09:44:11.247Z
631d760344503b7227837242
https://cdn-avatars.huggingface.co/v1/production/uploads/631d760344503b7227837242/3b6JRusFX6GKJpsN9ZdeJ.png
Max Ku
user
vinesmsuic
65fcf07da2d58888315a2e98
Cong Wei
admin_assigned
2024-03-22T09:41:19.908Z
64f8e358766ff9f3d2b0de84
https://cdn-avatars.huggingface.co/v1/production/uploads/64f8e358766ff9f3d2b0de84/R2P1YG-mRBh7TU9wkjGGk.jpeg
Cong Wei
user
CongWei1230
65fcf07da2d58888315a2e99
Weiming Ren
claimed_verified
2024-03-22T08:37:46.360Z
64405a9d518271b0d1beea38
/avatars/b702474588fd7090773320422417a582.svg
Weiming Ren
user
wren93
65fcf07da2d58888315a2e9a
Huan Yang
65fcf07da2d58888315a2e9b
Wenhu Chen
admin_assigned
2024-03-22T09:42:05.786Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
AnyV2V: A Plug-and-Play Framework For Any Video-to-Video Editing Tasks
Video-to-video editing involves editing a source video along with additional
control (such as text prompts, subjects, or styles) to generate a new video
that aligns with the source video and the provided control. Traditional methods
have been constrained to certain editing types, limiting their ability to meet
the wide range of user demands. In this paper, we introduce AnyV2V, a novel
training-free framework designed to simplify video editing into two primary
steps: (1) employing an off-the-shelf image editing model (e.g.
InstructPix2Pix, InstantID, etc) to modify the first frame, (2) utilizing an
existing image-to-video generation model (e.g. I2VGen-XL) for DDIM inversion
and feature injection. In the first stage, AnyV2V can plug in any existing
image editing tools to support an extensive array of video editing tasks.
Beyond the traditional prompt-based editing methods, AnyV2V also can support
novel video editing tasks, including reference-based style transfer,
subject-driven editing, and identity manipulation, which were unattainable by
previous methods. In the second stage, AnyV2V can plug in any existing
image-to-video models to perform DDIM inversion and intermediate feature
injection to maintain the appearance and motion consistency with the source
video. On the prompt-based editing, we show that AnyV2V can outperform the
previous best approach by 35\% on prompt alignment, and 25\% on human
preference. On the three novel tasks, we show that AnyV2V also achieves a high
success rate. We believe AnyV2V will continue to thrive due to its ability to
seamlessly integrate the fast-evolving image editing methods. Such
compatibility can help AnyV2V to increase its versatility to cater to diverse
user demands.
AnyV2V, a training-free framework, leverages existing image editing and generation models to perform a wide range of video editing tasks, including prompt-based editing and novel reference-based style transfer, with improved performance and versatility.
InstructPix2Pix InstantID I2VGen-XL DDIM inversion feature injection video editing image editing prompt-based editing reference-based style transfer subject-driven editing identity manipulation appearance consistency motion consistency