FreeLong: Training-Free Long Video Generation with SpectralBlend
  Temporal Attention
Video diffusion models have made substantial progress in various video
generation applications. However, training models for long video generation
tasks require significant computational and data resources, posing a challenge
to developing long video diffusion models. This paper investigates a
straightforward and training-free approach to extend an existing short video
diffusion model (e.g. pre-trained on 16-frame videos) for consistent long video
generation (e.g. 128 frames). Our preliminary observation has found that
directly applying the short video diffusion model to generate long videos can
lead to severe video quality degradation. Further investigation reveals that
this degradation is primarily due to the distortion of high-frequency
components in long videos, characterized by a decrease in spatial
high-frequency components and an increase in temporal high-frequency
components. Motivated by this, we propose a novel solution named FreeLong to
balance the frequency distribution of long video features during the denoising
process. FreeLong blends the low-frequency components of global video features,
which encapsulate the entire video sequence, with the high-frequency components
of local video features that focus on shorter subsequences of frames. This
approach maintains global consistency while incorporating diverse and
high-quality spatiotemporal details from local videos, enhancing both the
consistency and fidelity of long video generation. We evaluated FreeLong on
multiple base video diffusion models and observed significant improvements.
Additionally, our method supports coherent multi-prompt generation, ensuring
both visual coherence and seamless transitions between scenes.
66a850b261f6e00f69f6d730
Yu Lu
extracted_pending
2024-07-30T02:32:22.467Z
639ae93c3786549794e97c69
/avatars/65f7c0b641145f68f22072a5f77e086d.svg
YL
user
Simase
66a850b261f6e00f69f6d731
Yuanzhi Liang
66a850b261f6e00f69f6d732
Linchao Zhu
admin_assigned
2024-07-30T08:58:14.677Z
63521e1dfe367c0d9b155007
/avatars/b22804fc63b507fd60191486b17cdf7c.svg
Linchao Zhu
user
ffmpbgrnn
66a850b261f6e00f69f6d733
Yi Yang
FreeLong: Training-Free Long Video Generation with SpectralBlend
  Temporal Attention
Video diffusion models have made substantial progress in various video
generation applications. However, training models for long video generation
tasks require significant computational and data resources, posing a challenge
to developing long video diffusion models. This paper investigates a
straightforward and training-free approach to extend an existing short video
diffusion model (e.g. pre-trained on 16-frame videos) for consistent long video
generation (e.g. 128 frames). Our preliminary observation has found that
directly applying the short video diffusion model to generate long videos can
lead to severe video quality degradation. Further investigation reveals that
this degradation is primarily due to the distortion of high-frequency
components in long videos, characterized by a decrease in spatial
high-frequency components and an increase in temporal high-frequency
components. Motivated by this, we propose a novel solution named FreeLong to
balance the frequency distribution of long video features during the denoising
process. FreeLong blends the low-frequency components of global video features,
which encapsulate the entire video sequence, with the high-frequency components
of local video features that focus on shorter subsequences of frames. This
approach maintains global consistency while incorporating diverse and
high-quality spatiotemporal details from local videos, enhancing both the
consistency and fidelity of long video generation. We evaluated FreeLong on
multiple base video diffusion models and observed significant improvements.
Additionally, our method supports coherent multi-prompt generation, ensuring
both visual coherence and seamless transitions between scenes.
FreeLong enhances long video generation by balancing global and local video features during denoising, improving quality and consistency.
video diffusion models video generation long video generation short video diffusion model diffusion models temporal high-frequency components spatial high-frequency components denoising global video features local video features spatiotemporal details multi-prompt generation visual coherence seamless transitions