MOFA-Video: Controllable Image Animation via Generative Motion Field
  Adaptions in Frozen Image-to-Video Diffusion Model
We present MOFA-Video, an advanced controllable image animation method that
generates video from the given image using various additional controllable
signals (such as human landmarks reference, manual trajectories, and another
even provided video) or their combinations. This is different from previous
methods which only can work on a specific motion domain or show weak control
abilities with diffusion prior. To achieve our goal, we design several
domain-aware motion field adapters (\ie, MOFA-Adapters) to control the
generated motions in the video generation pipeline. For MOFA-Adapters, we
consider the temporal motion consistency of the video and generate the dense
motion flow from the given sparse control conditions first, and then, the
multi-scale features of the given image are wrapped as a guided feature for
stable video diffusion generation. We naively train two motion adapters for the
manual trajectories and the human landmarks individually since they both
contain sparse information about the control. After training, the MOFA-Adapters
in different domains can also work together for more controllable video
generation.
6659461d9ade6238ae865768
Muyao Niu
admin_assigned
2024-05-31T10:04:39.074Z
66597e53085eb651879aa4a1
/avatars/116aa003266bb19e8e03eb7ac5633b03.svg
Muyao Niu
user
MyNiuuu
6659461d9ade6238ae865769
Xiaodong Cun
admin_assigned
2024-05-31T10:04:44.551Z
63184c517ca1b876d99b7e0e
https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
Xiaodong Cun
user
vinthony
6659461d9ade6238ae86576a
Xintao Wang
admin_assigned
2024-05-31T10:04:50.430Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
6659461d9ade6238ae86576b
Yong Zhang
6659461d9ade6238ae86576c
Ying Shan
6659461d9ade6238ae86576d
Yinqiang Zheng
MOFA-Video: Controllable Image Animation via Generative Motion Field
  Adaptions in Frozen Image-to-Video Diffusion Model
We present MOFA-Video, an advanced controllable image animation method that
generates video from the given image using various additional controllable
signals (such as human landmarks reference, manual trajectories, and another
even provided video) or their combinations. This is different from previous
methods which only can work on a specific motion domain or show weak control
abilities with diffusion prior. To achieve our goal, we design several
domain-aware motion field adapters (\ie, MOFA-Adapters) to control the
generated motions in the video generation pipeline. For MOFA-Adapters, we
consider the temporal motion consistency of the video and generate the dense
motion flow from the given sparse control conditions first, and then, the
multi-scale features of the given image are wrapped as a guided feature for
stable video diffusion generation. We naively train two motion adapters for the
manual trajectories and the human landmarks individually since they both
contain sparse information about the control. After training, the MOFA-Adapters
in different domains can also work together for more controllable video
generation.
MOFA-Video generates videos from images using controllable signals, combining domain-aware motion field adapters to improve motion consistency and stability in diffusion-based video generation.
MOFA-Video image animation motion field adapters MOFA-Adapters temporal motion consistency dense motion flow sparse control conditions multi-scale features guided feature stable video diffusion generation