Video Motion Transfer with Diffusion Transformers
We propose DiTFlow, a method for transferring the motion of a reference video
to a newly synthesized one, designed specifically for Diffusion Transformers
(DiT). We first process the reference video with a pre-trained DiT to analyze
cross-frame attention maps and extract a patch-wise motion signal called the
Attention Motion Flow (AMF). We guide the latent denoising process in an
optimization-based, training-free, manner by optimizing latents with our AMF
loss to generate videos reproducing the motion of the reference one. We also
apply our optimization strategy to transformer positional embeddings, granting
us a boost in zero-shot motion transfer capabilities. We evaluate DiTFlow
against recently published methods, outperforming all across multiple metrics
and human evaluation.
67592b708ac3b988434f8261
Alexander Pondaven
claimed_verified
2024-12-11T09:19:44.410Z
63544cacb8aa3b0b3ebd9800
/avatars/7868b79742de27074376d5698e281003.svg
Alexander Pondaven
user
alexpondaven
67592b708ac3b988434f8262
Aliaksandr Siarohin
admin_assigned
2024-12-11T10:35:51.963Z
64276311eb9a0ed86180715b
/avatars/76f933cd549f10e5e2db379de235d304.svg
Aliaksandr Siarohin
user
aliaksandr-siarohin
67592b708ac3b988434f8263
Sergey Tulyakov
67592b708ac3b988434f8264
Philip Torr
admin_assigned
2024-12-11T10:36:23.031Z
6565ed28a5ec0231cb07225f
/avatars/7f95bba9aa7811d56eecb380827abfac.svg
prof philip torr
user
philiptorr
67592b708ac3b988434f8265
Fabio Pizzati
admin_assigned
2024-12-11T10:36:52.234Z
62bdcba92dd4214c0622ea30
/avatars/b62d86d01597473e05f9991ec04baf57.svg
Fabio Pizzati
user
fabvio
Video Motion Transfer with Diffusion Transformers
We propose DiTFlow, a method for transferring the motion of a reference video
to a newly synthesized one, designed specifically for Diffusion Transformers
(DiT). We first process the reference video with a pre-trained DiT to analyze
cross-frame attention maps and extract a patch-wise motion signal called the
Attention Motion Flow (AMF). We guide the latent denoising process in an
optimization-based, training-free, manner by optimizing latents with our AMF
loss to generate videos reproducing the motion of the reference one. We also
apply our optimization strategy to transformer positional embeddings, granting
us a boost in zero-shot motion transfer capabilities. We evaluate DiTFlow
against recently published methods, outperforming all across multiple metrics
and human evaluation.
Diffusion Transformers (DiT) Attention Motion Flow (AMF) latent denoising positional embeddings zero-shot motion transfer