Puppet-Master: Scaling Interactive Video Generation as a Motion Prior
  for Part-Level Dynamics
We present Puppet-Master, an interactive video generative model that can
serve as a motion prior for part-level dynamics. At test time, given a single
image and a sparse set of motion trajectories (i.e., drags), Puppet-Master can
synthesize a video depicting realistic part-level motion faithful to the given
drag interactions. This is achieved by fine-tuning a large-scale pre-trained
video diffusion model, for which we propose a new conditioning architecture to
inject the dragging control effectively. More importantly, we introduce the
all-to-first attention mechanism, a drop-in replacement for the widely adopted
spatial attention modules, which significantly improves generation quality by
addressing the appearance and background issues in existing models. Unlike
other motion-conditioned video generators that are trained on in-the-wild
videos and mostly move an entire object, Puppet-Master is learned from
Objaverse-Animation-HQ, a new dataset of curated part-level motion clips. We
propose a strategy to automatically filter out sub-optimal animations and
augment the synthetic renderings with meaningful motion trajectories.
Puppet-Master generalizes well to real images across various categories and
outperforms existing methods in a zero-shot manner on a real-world benchmark.
See our project page for more results: vgg-puppetmaster.github.io.
66b5756218d95c926cb7132e
Ruining Li
admin_assigned
2024-08-09T10:22:06.290Z
6467cc17e7a6a374fd1a41e5
/avatars/3cbd4ad5bc17067ee38b13d0cbadd80c.svg
Ruining Li
user
rayli
66b5756218d95c926cb7132f
Chuanxia Zheng
66b5756218d95c926cb71330
Christian Rupprecht
66b5756218d95c926cb71331
Andrea Vedaldi
Puppet-Master: Scaling Interactive Video Generation as a Motion Prior
  for Part-Level Dynamics
We present Puppet-Master, an interactive video generative model that can
serve as a motion prior for part-level dynamics. At test time, given a single
image and a sparse set of motion trajectories (i.e., drags), Puppet-Master can
synthesize a video depicting realistic part-level motion faithful to the given
drag interactions. This is achieved by fine-tuning a large-scale pre-trained
video diffusion model, for which we propose a new conditioning architecture to
inject the dragging control effectively. More importantly, we introduce the
all-to-first attention mechanism, a drop-in replacement for the widely adopted
spatial attention modules, which significantly improves generation quality by
addressing the appearance and background issues in existing models. Unlike
other motion-conditioned video generators that are trained on in-the-wild
videos and mostly move an entire object, Puppet-Master is learned from
Objaverse-Animation-HQ, a new dataset of curated part-level motion clips. We
propose a strategy to automatically filter out sub-optimal animations and
augment the synthetic renderings with meaningful motion trajectories.
Puppet-Master generalizes well to real images across various categories and
outperforms existing methods in a zero-shot manner on a real-world benchmark.
See our project page for more results: vgg-puppetmaster.github.io.
Puppet-Master is an interactive video generative model that uses fine-tuned diffusion with part-level motion and all-to-first attention, creating realistic videos based on sparse motion control.
diffusion models conditioning architecture all-to-first attention motion-conditioned video generators Objaverse-Animation-HQ dataset zero-shot performance