DragNUWA: Fine-grained Control in Video Generation by Integrating Text,
  Image, and Trajectory
Controllable video generation has gained significant attention in recent
years. However, two main limitations persist: Firstly, most existing works
focus on either text, image, or trajectory-based control, leading to an
inability to achieve fine-grained control in videos. Secondly, trajectory
control research is still in its early stages, with most experiments being
conducted on simple datasets like Human3.6M. This constraint limits the models'
capability to process open-domain images and effectively handle complex curved
trajectories. In this paper, we propose DragNUWA, an open-domain
diffusion-based video generation model. To tackle the issue of insufficient
control granularity in existing works, we simultaneously introduce text, image,
and trajectory information to provide fine-grained control over video content
from semantic, spatial, and temporal perspectives. To resolve the problem of
limited open-domain trajectory control in current research, We propose
trajectory modeling with three aspects: a Trajectory Sampler (TS) to enable
open-domain control of arbitrary trajectories, a Multiscale Fusion (MF) to
control trajectories in different granularities, and an Adaptive Training (AT)
strategy to generate consistent videos following trajectories. Our experiments
validate the effectiveness of DragNUWA, demonstrating its superior performance
in fine-grained control in video generation. The homepage link is
https://www.microsoft.com/en-us/research/project/dragnuwa/
64ddbe671a14509152994239
Shengming Yin
admin_assigned
2023-08-17T14:11:05.764Z
62430b00b1fc6718f4fa6295
/avatars/f4554c124ec12ff0457598b5db4b175b.svg
LanHarmony
user
LanHarmony
64ddbe671a1450915299423a
Chenfei Wu
admin_assigned
2023-08-17T14:11:18.425Z
63180f0f49af4d408c35f07f
https://cdn-avatars.huggingface.co/v1/production/uploads/1678864054390-63180f0f49af4d408c35f07f.jpeg
Chenfei Wu
user
Chenfei
64ddbe671a1450915299423b
Jian Liang
64ddbe671a1450915299423c
Jie Shi
64ddbe671a1450915299423d
Houqiang Li
64ddbe671a1450915299423e
Gong Ming
64ddbe671a1450915299423f
Nan Duan
DragNUWA: Fine-grained Control in Video Generation by Integrating Text,
  Image, and Trajectory
Controllable video generation has gained significant attention in recent
years. However, two main limitations persist: Firstly, most existing works
focus on either text, image, or trajectory-based control, leading to an
inability to achieve fine-grained control in videos. Secondly, trajectory
control research is still in its early stages, with most experiments being
conducted on simple datasets like Human3.6M. This constraint limits the models'
capability to process open-domain images and effectively handle complex curved
trajectories. In this paper, we propose DragNUWA, an open-domain
diffusion-based video generation model. To tackle the issue of insufficient
control granularity in existing works, we simultaneously introduce text, image,
and trajectory information to provide fine-grained control over video content
from semantic, spatial, and temporal perspectives. To resolve the problem of
limited open-domain trajectory control in current research, We propose
trajectory modeling with three aspects: a Trajectory Sampler (TS) to enable
open-domain control of arbitrary trajectories, a Multiscale Fusion (MF) to
control trajectories in different granularities, and an Adaptive Training (AT)
strategy to generate consistent videos following trajectories. Our experiments
validate the effectiveness of DragNUWA, demonstrating its superior performance
in fine-grained control in video generation. The homepage link is
https://www.microsoft.com/en-us/research/project/dragnuwa/
DragNUWA is a diffusion-based video generation model that integrates text, image, and trajectory controls for fine-grained and open-domain video creation.
controllable video generation diffusion-based video generation text-based control image-based control trajectory-based control Trajectory Sampler Multiscale Fusion Adaptive Training