Generative Inbetweening: Adapting Image-to-Video Models for Keyframe
  Interpolation
We present a method for generating video sequences with coherent motion
between a pair of input key frames. We adapt a pretrained large-scale
image-to-video diffusion model (originally trained to generate videos moving
forward in time from a single input image) for key frame interpolation, i.e.,
to produce a video in between two input frames. We accomplish this adaptation
through a lightweight fine-tuning technique that produces a version of the
model that instead predicts videos moving backwards in time from a single input
image. This model (along with the original forward-moving model) is
subsequently used in a dual-directional diffusion sampling process that
combines the overlapping model estimates starting from each of the two
keyframes. Our experiments show that our method outperforms both existing
diffusion-based methods and traditional frame interpolation techniques.
66ce96ed6116c7f53e64c579
Xiaojuan Wang
admin_assigned
2024-08-28T08:26:25.939Z
643ee9a6022a0350efd745d1
/avatars/c8c17689ed88b1966d2a36361e3f83cc.svg
Xiaojuan Wang
user
xiaojwan
66ce96ed6116c7f53e64c57a
Boyang Zhou
admin_assigned
2024-08-28T08:26:33.427Z
6445f1105691ca69b0df48fc
/avatars/0e2fc769a0c044d0dd9a9c0debf49782.svg
Boyang Zhou
user
BobbyZ963
66ce96ed6116c7f53e64c57b
Brian Curless
66ce96ed6116c7f53e64c57c
Ira Kemelmacher-Shlizerman
claimed_verified
2024-09-02T07:19:03.265Z
66d20e411ba71ac4c0488132
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/kkqDAFT15lx4wrJVXiFGJ.png
Ira Kemelmacher-Shlizerman
user
kemelmi
66ce96ed6116c7f53e64c57d
Aleksander Holynski
admin_assigned
2024-08-28T08:27:26.657Z
62f6e2792e53c2efd33faa92
/avatars/85d94b4022577747b8d2d10a82c2f3c7.svg
Aleksander Holynski
user
holynski
66ce96ed6116c7f53e64c57e
Steven M. Seitz
Generative Inbetweening: Adapting Image-to-Video Models for Keyframe
  Interpolation
We present a method for generating video sequences with coherent motion
between a pair of input key frames. We adapt a pretrained large-scale
image-to-video diffusion model (originally trained to generate videos moving
forward in time from a single input image) for key frame interpolation, i.e.,
to produce a video in between two input frames. We accomplish this adaptation
through a lightweight fine-tuning technique that produces a version of the
model that instead predicts videos moving backwards in time from a single input
image. This model (along with the original forward-moving model) is
subsequently used in a dual-directional diffusion sampling process that
combines the overlapping model estimates starting from each of the two
keyframes. Our experiments show that our method outperforms both existing
diffusion-based methods and traditional frame interpolation techniques.
A method for generating coherent video sequences between key frames using a dual-directional diffusion process outperforms existing techniques.
diffusion model key frame interpolation fine-tuning dual-directional diffusion sampling video generation