Motion-I2V: Consistent and Controllable Image-to-Video Generation with
  Explicit Motion Modeling
We introduce Motion-I2V, a novel framework for consistent and controllable
image-to-video generation (I2V). In contrast to previous methods that directly
learn the complicated image-to-video mapping, Motion-I2V factorizes I2V into
two stages with explicit motion modeling. For the first stage, we propose a
diffusion-based motion field predictor, which focuses on deducing the
trajectories of the reference image's pixels. For the second stage, we propose
motion-augmented temporal attention to enhance the limited 1-D temporal
attention in video latent diffusion models. This module can effectively
propagate reference image's feature to synthesized frames with the guidance of
predicted trajectories from the first stage. Compared with existing methods,
Motion-I2V can generate more consistent videos even at the presence of large
motion and viewpoint variation. By training a sparse trajectory ControlNet for
the first stage, Motion-I2V can support users to precisely control motion
trajectories and motion regions with sparse trajectory and region annotations.
This offers more controllability of the I2V process than solely relying on
textual instructions. Additionally, Motion-I2V's second stage naturally
supports zero-shot video-to-video translation. Both qualitative and
quantitative comparisons demonstrate the advantages of Motion-I2V over prior
approaches in consistent and controllable image-to-video generation.
65b8611d6169e07d8d7bc979
Xiaoyu Shi
65b8611d6169e07d8d7bc97a
Zhaoyang Huang
admin_assigned
2024-01-30T09:32:52.442Z
64c26e0e400274d129a10d5d
/avatars/c1debf3d7f61c2ea01d9a25d7e332712.svg
Zhaoyang Huang
user
DrinkingCoder
65b8611d6169e07d8d7bc97b
Fu-Yun Wang
claimed_verified
2024-01-30T08:06:41.271Z
63e9e92f20c109718713f5eb
/avatars/9ff312e854d803e1a2e9e685a21d12f8.svg
Fu-Yun Wang
user
wangfuyun
65b8611d6169e07d8d7bc97c
Weikang Bian
claimed_verified
2024-01-30T09:00:01.707Z
65b8b840a81930c593073370
/avatars/31b399d3da33e8c02f9e06a16dd85779.svg
Weikang BIAN
user
wkbian
65b8611d6169e07d8d7bc97d
Dasong Li
65b8611d6169e07d8d7bc97e
Yi Zhang
65b8611d6169e07d8d7bc97f
Manyuan Zhang
65b8611d6169e07d8d7bc980
Ka Chun Cheung
65b8611d6169e07d8d7bc981
Simon See
65b8611d6169e07d8d7bc982
Hongwei Qin
65b8611d6169e07d8d7bc983
Jifeng Da
admin_assigned
2024-01-30T09:37:55.235Z
64686f7172d9180d4ac8b4e4
/avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
Jifeng Dai
user
daijifeng
65b8611d6169e07d8d7bc984
Hongsheng Li
claimed_verified
2024-02-05T10:49:23.281Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
Motion-I2V: Consistent and Controllable Image-to-Video Generation with
  Explicit Motion Modeling
We introduce Motion-I2V, a novel framework for consistent and controllable
image-to-video generation (I2V). In contrast to previous methods that directly
learn the complicated image-to-video mapping, Motion-I2V factorizes I2V into
two stages with explicit motion modeling. For the first stage, we propose a
diffusion-based motion field predictor, which focuses on deducing the
trajectories of the reference image's pixels. For the second stage, we propose
motion-augmented temporal attention to enhance the limited 1-D temporal
attention in video latent diffusion models. This module can effectively
propagate reference image's feature to synthesized frames with the guidance of
predicted trajectories from the first stage. Compared with existing methods,
Motion-I2V can generate more consistent videos even at the presence of large
motion and viewpoint variation. By training a sparse trajectory ControlNet for
the first stage, Motion-I2V can support users to precisely control motion
trajectories and motion regions with sparse trajectory and region annotations.
This offers more controllability of the I2V process than solely relying on
textual instructions. Additionally, Motion-I2V's second stage naturally
supports zero-shot video-to-video translation. Both qualitative and
quantitative comparisons demonstrate the advantages of Motion-I2V over prior
approaches in consistent and controllable image-to-video generation.
Motion-I2V, a diffusion-based framework for image-to-video generation, achieves consistency and controllability through explicit motion modeling and trajectory prediction.
diffusion-based motion field predictor motion-augmented temporal attention video latent diffusion models sparse trajectory ControlNet zero-shot video-to-video translation