VidToMe: Video Token Merging for Zero-Shot Video Editing
Diffusion models have made significant advances in generating high-quality
images, but their application to video generation has remained challenging due
to the complexity of temporal motion. Zero-shot video editing offers a solution
by utilizing pre-trained image diffusion models to translate source videos into
new ones. Nevertheless, existing methods struggle to maintain strict temporal
consistency and efficient memory consumption. In this work, we propose a novel
approach to enhance temporal consistency in generated videos by merging
self-attention tokens across frames. By aligning and compressing temporally
redundant tokens across frames, our method improves temporal coherence and
reduces memory consumption in self-attention computations. The merging strategy
matches and aligns tokens according to the temporal correspondence between
frames, facilitating natural temporal consistency in generated video frames. To
manage the complexity of video processing, we divide videos into chunks and
develop intra-chunk local token merging and inter-chunk global token merging,
ensuring both short-term video continuity and long-term content consistency.
Our video editing approach seamlessly extends the advancements in image editing
to video editing, rendering favorable results in temporal consistency over
state-of-the-art methods.
6581287a23a7aac397d674eb
Xirui Li
admin_assigned
2023-12-19T10:08:05.579Z
6451364c4b84eee0deea68fa
/avatars/cc5c52f9b30f43b7faa47d0d5848a492.svg
Xirui Li
user
lixirui142
6581287a23a7aac397d674ec
Chao Ma
6581287a23a7aac397d674ed
Xiaokang Yang
admin_assigned
2023-12-19T10:08:56.434Z
64eaba6b1d8b651a9aed1203
/avatars/a0f26095c262ca1a204c27ff4bfb188c.svg
yangxiaokang
user
zicheqingluo
6581287a23a7aac397d674ee
Ming-Hsuan Yang
VidToMe: Video Token Merging for Zero-Shot Video Editing
Diffusion models have made significant advances in generating high-quality
images, but their application to video generation has remained challenging due
to the complexity of temporal motion. Zero-shot video editing offers a solution
by utilizing pre-trained image diffusion models to translate source videos into
new ones. Nevertheless, existing methods struggle to maintain strict temporal
consistency and efficient memory consumption. In this work, we propose a novel
approach to enhance temporal consistency in generated videos by merging
self-attention tokens across frames. By aligning and compressing temporally
redundant tokens across frames, our method improves temporal coherence and
reduces memory consumption in self-attention computations. The merging strategy
matches and aligns tokens according to the temporal correspondence between
frames, facilitating natural temporal consistency in generated video frames. To
manage the complexity of video processing, we divide videos into chunks and
develop intra-chunk local token merging and inter-chunk global token merging,
ensuring both short-term video continuity and long-term content consistency.
Our video editing approach seamlessly extends the advancements in image editing
to video editing, rendering favorable results in temporal consistency over
state-of-the-art methods.
A novel method improves temporal consistency and reduces memory consumption in video generated by diffusion models through token merging across frames.
diffusion models zero-shot video editing self-attention temporal consistency token merging temporal coherence intra-chunk local token merging inter-chunk global token merging short-term video continuity long-term content consistency