ViBiDSampler: Enhancing Video Interpolation Using Bidirectional
  Diffusion Sampler
Recent progress in large-scale text-to-video (T2V) and image-to-video (I2V)
diffusion models has greatly enhanced video generation, especially in terms of
keyframe interpolation. However, current image-to-video diffusion models, while
powerful in generating videos from a single conditioning frame, need adaptation
for two-frame (start & end) conditioned generation, which is essential for
effective bounded interpolation. Unfortunately, existing approaches that fuse
temporally forward and backward paths in parallel often suffer from
off-manifold issues, leading to artifacts or requiring multiple iterative
re-noising steps. In this work, we introduce a novel, bidirectional sampling
strategy to address these off-manifold issues without requiring extensive
re-noising or fine-tuning. Our method employs sequential sampling along both
forward and backward paths, conditioned on the start and end frames,
respectively, ensuring more coherent and on-manifold generation of intermediate
frames. Additionally, we incorporate advanced guidance techniques, CFG++ and
DDS, to further enhance the interpolation process. By integrating these, our
method achieves state-of-the-art performance, efficiently generating
high-quality, smooth videos between keyframes. On a single 3090 GPU, our method
can interpolate 25 frames at 1024 x 576 resolution in just 195 seconds,
establishing it as a leading solution for keyframe interpolation.
6707511a95f90bdb8e8dfc5e
Serin Yang
admin_assigned
2024-10-10T09:49:23.096Z
63a2b7f9cc214062058e5a81
/avatars/be49a6e3aa6e4a3ad8800858d9a82ef1.svg
Serin Yang
user
sr2851766
6707511a95f90bdb8e8dfc5f
Taesung Kwon
6707511a95f90bdb8e8dfc60
Jong Chul Ye
ViBiDSampler: Enhancing Video Interpolation Using Bidirectional
  Diffusion Sampler
Recent progress in large-scale text-to-video (T2V) and image-to-video (I2V)
diffusion models has greatly enhanced video generation, especially in terms of
keyframe interpolation. However, current image-to-video diffusion models, while
powerful in generating videos from a single conditioning frame, need adaptation
for two-frame (start & end) conditioned generation, which is essential for
effective bounded interpolation. Unfortunately, existing approaches that fuse
temporally forward and backward paths in parallel often suffer from
off-manifold issues, leading to artifacts or requiring multiple iterative
re-noising steps. In this work, we introduce a novel, bidirectional sampling
strategy to address these off-manifold issues without requiring extensive
re-noising or fine-tuning. Our method employs sequential sampling along both
forward and backward paths, conditioned on the start and end frames,
respectively, ensuring more coherent and on-manifold generation of intermediate
frames. Additionally, we incorporate advanced guidance techniques, CFG++ and
DDS, to further enhance the interpolation process. By integrating these, our
method achieves state-of-the-art performance, efficiently generating
high-quality, smooth videos between keyframes. On a single 3090 GPU, our method
can interpolate 25 frames at 1024 x 576 resolution in just 195 seconds,
establishing it as a leading solution for keyframe interpolation.
A bidirectional sampling strategy combined with CFG++ and DDS is introduced to improve image-to-video diffusion models for effective keyframe interpolation, achieving high-quality results quickly.
text-to-video image-to-video diffusion models keyframe interpolation bidirectional sampling off-manifold issues CFG++ DDS intermediate frames