Motion Prompting: Controlling Video Generation with Motion Trajectories
Motion control is crucial for generating expressive and compelling video
content; however, most existing video generation models rely mainly on text
prompts for control, which struggle to capture the nuances of dynamic actions
and temporal compositions. To this end, we train a video generation model
conditioned on spatio-temporally sparse or dense motion trajectories. In
contrast to prior motion conditioning work, this flexible representation can
encode any number of trajectories, object-specific or global scene motion, and
temporally sparse motion; due to its flexibility we refer to this conditioning
as motion prompts. While users may directly specify sparse trajectories, we
also show how to translate high-level user requests into detailed, semi-dense
motion prompts, a process we term motion prompt expansion. We demonstrate the
versatility of our approach through various applications, including camera and
object motion control, "interacting" with an image, motion transfer, and image
editing. Our results showcase emergent behaviors, such as realistic physics,
suggesting the potential of motion prompts for probing video models and
interacting with future generative world models. Finally, we evaluate
quantitatively, conduct a human study, and demonstrate strong performance.
Video results are available on our webpage: https://motion-prompting.github.io/
6750808b94f48729d536d9d1
Daniel Geng
6750808b94f48729d536d9d2
Charles Herrmann
admin_assigned
2024-12-16T13:58:58.267Z
64c7f088c864d962eddeaf8c
/avatars/e6364c56e6cb6907b6dd2d34eed83720.svg
Charles Herrmann
user
irwinherrmann
6750808b94f48729d536d9d3
Junhwa Hur
admin_assigned
2024-12-16T13:58:50.219Z
647f3046b0e96764589e107a
/avatars/1074ee5ec1b2d5d297406f9d455213c4.svg
Junhwa Hur
user
hurjunhwa
6750808b94f48729d536d9d4
Forrester Cole
admin_assigned
2024-12-16T13:58:42.494Z
635003011d81beb8e244e68d
/avatars/32f46e6778d1c52ebdd6f2511ba48257.svg
Forrester Cole
user
fcole
6750808b94f48729d536d9d5
Serena Zhang
admin_assigned
2024-12-16T13:58:36.556Z
64b817e4cf14c2fabe85ef68
/avatars/88093121b07adbc14c53c504c67709ea.svg
Serena Zhang
user
serenaz
6750808b94f48729d536d9d6
Tobias Pfaff
6750808b94f48729d536d9d7
Tatiana Lopez-Guevara
6750808b94f48729d536d9d8
Carl Doersch
6750808b94f48729d536d9d9
Yusuf Aytar
6750808b94f48729d536d9da
Michael Rubinstein
6750808b94f48729d536d9db
Chen Sun
6750808b94f48729d536d9dc
Oliver Wang
6750808b94f48729d536d9dd
Andrew Owens
6750808b94f48729d536d9de
Deqing Sun
admin_assigned
2024-12-16T13:58:06.713Z
6750994d0438fdc8b9f86303
/avatars/fe89e72a6cd15649df2c7845033fa8ae.svg
Deqing Sun
user
deqingsun
Motion Prompting: Controlling Video Generation with Motion Trajectories
Motion control is crucial for generating expressive and compelling video
content; however, most existing video generation models rely mainly on text
prompts for control, which struggle to capture the nuances of dynamic actions
and temporal compositions. To this end, we train a video generation model
conditioned on spatio-temporally sparse or dense motion trajectories. In
contrast to prior motion conditioning work, this flexible representation can
encode any number of trajectories, object-specific or global scene motion, and
temporally sparse motion; due to its flexibility we refer to this conditioning
as motion prompts. While users may directly specify sparse trajectories, we
also show how to translate high-level user requests into detailed, semi-dense
motion prompts, a process we term motion prompt expansion. We demonstrate the
versatility of our approach through various applications, including camera and
object motion control, "interacting" with an image, motion transfer, and image
editing. Our results showcase emergent behaviors, such as realistic physics,
suggesting the potential of motion prompts for probing video models and
interacting with future generative world models. Finally, we evaluate
quantitatively, conduct a human study, and demonstrate strong performance.
Video results are available on our webpage: https://motion-prompting.github.io/
A video generation model conditioned on motion prompts can capture dynamic actions and temporal compositions, supporting various applications like camera control and motion transfer with realistic physics.
motion prompts spatio-temporally sparse spatio-temporally dense motion trajectories object-specific motion global scene motion temporally sparse motion motion prompt expansion video generation model