Motion Control for Enhanced Complex Action Video Generation
Existing text-to-video (T2V) models often struggle with generating videos
with sufficiently pronounced or complex actions. A key limitation lies in the
text prompt's inability to precisely convey intricate motion details. To
address this, we propose a novel framework, MVideo, designed to produce
long-duration videos with precise, fluid actions. MVideo overcomes the
limitations of text prompts by incorporating mask sequences as an additional
motion condition input, providing a clearer, more accurate representation of
intended actions. Leveraging foundational vision models such as GroundingDINO
and SAM2, MVideo automatically generates mask sequences, enhancing both
efficiency and robustness. Our results demonstrate that, after training, MVideo
effectively aligns text prompts with motion conditions to produce videos that
simultaneously meet both criteria. This dual control mechanism allows for more
dynamic video generation by enabling alterations to either the text prompt or
motion condition independently, or both in tandem. Furthermore, MVideo supports
motion condition editing and composition, facilitating the generation of videos
with more complex actions. MVideo thus advances T2V motion generation, setting
a strong benchmark for improved action depiction in current video diffusion
models. Our project page is available at https://mvideo-v1.github.io/.
6736121f12b0cbbb1232b84f
Qiang Zhou
claimed_verified
2024-11-15T14:33:25.630Z
64db1edc7266618e853c6c13
/avatars/b3362a882fd9cee3d9f59d8dbcc02367.svg
mightyzau
user
mightyzau
6736121f12b0cbbb1232b850
Shaofeng Zhang
admin_assigned
2024-11-15T14:24:41.026Z
669912b8a22692f3544c8ec2
/avatars/3e4f90b4bf9c0e6e8dcf0bc8bd9d8cfa.svg
Shaofeng Zhang
user
SimonZ1881
6736121f12b0cbbb1232b851
Nianzu Yang
admin_assigned
2024-11-15T14:24:47.634Z
66371a41164dfca129939a72
/avatars/2fdf0a059e06643f9f92c57efd1610b5.svg
Nianzu Yang
user
yangnianzu
6736121f12b0cbbb1232b852
Ye Qian
6736121f12b0cbbb1232b853
Hao Li
Motion Control for Enhanced Complex Action Video Generation
Existing text-to-video (T2V) models often struggle with generating videos
with sufficiently pronounced or complex actions. A key limitation lies in the
text prompt's inability to precisely convey intricate motion details. To
address this, we propose a novel framework, MVideo, designed to produce
long-duration videos with precise, fluid actions. MVideo overcomes the
limitations of text prompts by incorporating mask sequences as an additional
motion condition input, providing a clearer, more accurate representation of
intended actions. Leveraging foundational vision models such as GroundingDINO
and SAM2, MVideo automatically generates mask sequences, enhancing both
efficiency and robustness. Our results demonstrate that, after training, MVideo
effectively aligns text prompts with motion conditions to produce videos that
simultaneously meet both criteria. This dual control mechanism allows for more
dynamic video generation by enabling alterations to either the text prompt or
motion condition independently, or both in tandem. Furthermore, MVideo supports
motion condition editing and composition, facilitating the generation of videos
with more complex actions. MVideo thus advances T2V motion generation, setting
a strong benchmark for improved action depiction in current video diffusion
models. Our project page is available at https://mvideo-v1.github.io/.
MVideo enhances text-to-video generation by incorporating mask sequences for precise action control, improving the depiction of complex motions in long-duration videos.
text-to-video MVideo mask sequences motion condition GroundingDINO SAM2 video diffusion models