FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation
The remarkable efficacy of text-to-image diffusion models has motivated
extensive exploration of their potential application in video domains.
Zero-shot methods seek to extend image diffusion models to videos without
necessitating model training. Recent methods mainly focus on incorporating
inter-frame correspondence into attention mechanisms. However, the soft
constraint imposed on determining where to attend to valid features can
sometimes be insufficient, resulting in temporal inconsistency. In this paper,
we introduce FRESCO, intra-frame correspondence alongside inter-frame
correspondence to establish a more robust spatial-temporal constraint. This
enhancement ensures a more consistent transformation of semantically similar
content across frames. Beyond mere attention guidance, our approach involves an
explicit update of features to achieve high spatial-temporal consistency with
the input video, significantly improving the visual coherence of the resulting
translated videos. Extensive experiments demonstrate the effectiveness of our
proposed framework in producing high-quality, coherent videos, marking a
notable improvement over existing zero-shot methods.
65fa679becb85d8ad7e030c2
Shuai Yang
admin_assigned
2024-03-21T13:15:19.551Z
62a54d0410334c1d024e2f59
https://cdn-avatars.huggingface.co/v1/production/uploads/1664764278226-62a54d0410334c1d024e2f59.jpeg
Shuai Yang
user
PKUWilliamYang
65fa679becb85d8ad7e030c3
Yifan Zhou
claimed_verified
2024-12-02T12:52:46.241Z
64a63f9449b08110f761cd73
/avatars/61860202fc818b105ef24e74dd4f7d3c.svg
Yifan Zhou
user
SingleZombie
65fa679becb85d8ad7e030c4
Ziwei Liu
admin_assigned
2024-03-21T13:15:59.554Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
65fa679becb85d8ad7e030c5
Chen Change Loy
FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation
The remarkable efficacy of text-to-image diffusion models has motivated
extensive exploration of their potential application in video domains.
Zero-shot methods seek to extend image diffusion models to videos without
necessitating model training. Recent methods mainly focus on incorporating
inter-frame correspondence into attention mechanisms. However, the soft
constraint imposed on determining where to attend to valid features can
sometimes be insufficient, resulting in temporal inconsistency. In this paper,
we introduce FRESCO, intra-frame correspondence alongside inter-frame
correspondence to establish a more robust spatial-temporal constraint. This
enhancement ensures a more consistent transformation of semantically similar
content across frames. Beyond mere attention guidance, our approach involves an
explicit update of features to achieve high spatial-temporal consistency with
the input video, significantly improving the visual coherence of the resulting
translated videos. Extensive experiments demonstrate the effectiveness of our
proposed framework in producing high-quality, coherent videos, marking a
notable improvement over existing zero-shot methods.
FRESCO enhances zero-shot text-to-image diffusion models for video by integrating intra-frame and inter-frame correspondence, leading to improved spatial-temporal consistency and video coherence.
text-to-image diffusion models zero-shot methods inter-frame correspondence attention mechanisms intra-frame correspondence spatial-temporal constraint feature update spatial-temporal consistency visual coherence