FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video
  Synthesis
Diffusion models have transformed the image-to-image (I2I) synthesis and are
now permeating into videos. However, the advancement of video-to-video (V2V)
synthesis has been hampered by the challenge of maintaining temporal
consistency across video frames. This paper proposes a consistent V2V synthesis
framework by jointly leveraging spatial conditions and temporal optical flow
clues within the source video. Contrary to prior methods that strictly adhere
to optical flow, our approach harnesses its benefits while handling the
imperfection in flow estimation. We encode the optical flow via warping from
the first frame and serve it as a supplementary reference in the diffusion
model. This enables our model for video synthesis by editing the first frame
with any prevalent I2I models and then propagating edits to successive frames.
Our V2V model, FlowVid, demonstrates remarkable properties: (1) Flexibility:
FlowVid works seamlessly with existing I2I models, facilitating various
modifications, including stylization, object swaps, and local edits. (2)
Efficiency: Generation of a 4-second video with 30 FPS and 512x512 resolution
takes only 1.5 minutes, which is 3.1x, 7.2x, and 10.5x faster than CoDeF,
Rerender, and TokenFlow, respectively. (3) High-quality: In user studies, our
FlowVid is preferred 45.7% of the time, outperforming CoDeF (3.5%), Rerender
(10.2%), and TokenFlow (40.4%).
659234bd11f68f12ea863474
Feng Liang
claimed_verified
2024-01-02T09:27:51.577Z
63631be58f43a912fc7595d5
https://cdn-avatars.huggingface.co/v1/production/uploads/1667440137801-63631be58f43a912fc7595d5.jpeg
Jeff Liang
user
JeffLiang
659234bd11f68f12ea863475
Bichen Wu
659234bd11f68f12ea863476
Jialiang Wang
659234bd11f68f12ea863477
Licheng Yu
659234bd11f68f12ea863478
Kunpeng Li
659234bd11f68f12ea863479
Yinan Zhao
659234bd11f68f12ea86347a
Ishan Misra
659234bd11f68f12ea86347b
Jia-Bin Huang
659234bd11f68f12ea86347c
Peizhao Zhang
659234bd11f68f12ea86347d
Peter Vajda
659234bd11f68f12ea86347e
Diana Marculescu
FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video
  Synthesis
Diffusion models have transformed the image-to-image (I2I) synthesis and are
now permeating into videos. However, the advancement of video-to-video (V2V)
synthesis has been hampered by the challenge of maintaining temporal
consistency across video frames. This paper proposes a consistent V2V synthesis
framework by jointly leveraging spatial conditions and temporal optical flow
clues within the source video. Contrary to prior methods that strictly adhere
to optical flow, our approach harnesses its benefits while handling the
imperfection in flow estimation. We encode the optical flow via warping from
the first frame and serve it as a supplementary reference in the diffusion
model. This enables our model for video synthesis by editing the first frame
with any prevalent I2I models and then propagating edits to successive frames.
Our V2V model, FlowVid, demonstrates remarkable properties: (1) Flexibility:
FlowVid works seamlessly with existing I2I models, facilitating various
modifications, including stylization, object swaps, and local edits. (2)
Efficiency: Generation of a 4-second video with 30 FPS and 512x512 resolution
takes only 1.5 minutes, which is 3.1x, 7.2x, and 10.5x faster than CoDeF,
Rerender, and TokenFlow, respectively. (3) High-quality: In user studies, our
FlowVid is preferred 45.7% of the time, outperforming CoDeF (3.5%), Rerender
(10.2%), and TokenFlow (40.4%).
A proposed consistent V2V synthesis framework uses diffusion models combined with encoded optical flow to propagate edits across video frames, offering flexibility, efficiency, and high-quality results.
diffusion models video-to-video (V2V) synthesis temporal optical flow spatial conditions warping FlowVid