Video Creation by Demonstration
We explore a novel video creation experience, namely Video Creation by
Demonstration. Given a demonstration video and a context image from a different
scene, we generate a physically plausible video that continues naturally from
the context image and carries out the action concepts from the demonstration.
To enable this capability, we present delta-Diffusion, a self-supervised
training approach that learns from unlabeled videos by conditional future frame
prediction. Unlike most existing video generation controls that are based on
explicit signals, we adopts the form of implicit latent control for maximal
flexibility and expressiveness required by general videos. By leveraging a
video foundation model with an appearance bottleneck design on top, we extract
action latents from demonstration videos for conditioning the generation
process with minimal appearance leakage. Empirically, delta-Diffusion
outperforms related baselines in terms of both human preference and large-scale
machine evaluations, and demonstrates potentials towards interactive world
simulation. Sampled video generation results are available at
https://delta-diffusion.github.io/.
675b963adbb389af011b5a01
Yihong Sun
675b963adbb389af011b5a02
Hao Zhou
675b963adbb389af011b5a03
Liangzhe Yuan
675b963adbb389af011b5a04
Jennifer J. Sun
675b963adbb389af011b5a05
Yandong Li
675b963adbb389af011b5a06
Xuhui Jia
675b963adbb389af011b5a07
Hartwig Adam
675b963adbb389af011b5a08
Bharath Hariharan
675b963adbb389af011b5a09
Long Zhao
extracted_confirmed
2024-12-13T02:13:07.970Z
650c249887dcda6616baa040
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tIVfxoAHAJ0sWyDMkaarA.jpeg
Long Zhao
user
garyzhao9012
675b963adbb389af011b5a0a
Ting Liu
claimed_verified
2024-12-13T15:22:35.393Z
64a78e9bacc69c3bbaf12b3d
/avatars/09cd26234e0e0c82f5cc2c02e710163c.svg
Ting Liu
user
tingianliu
Video Creation by Demonstration
We explore a novel video creation experience, namely Video Creation by
Demonstration. Given a demonstration video and a context image from a different
scene, we generate a physically plausible video that continues naturally from
the context image and carries out the action concepts from the demonstration.
To enable this capability, we present delta-Diffusion, a self-supervised
training approach that learns from unlabeled videos by conditional future frame
prediction. Unlike most existing video generation controls that are based on
explicit signals, we adopts the form of implicit latent control for maximal
flexibility and expressiveness required by general videos. By leveraging a
video foundation model with an appearance bottleneck design on top, we extract
action latents from demonstration videos for conditioning the generation
process with minimal appearance leakage. Empirically, delta-Diffusion
outperforms related baselines in terms of both human preference and large-scale
machine evaluations, and demonstrates potentials towards interactive world
simulation. Sampled video generation results are available at
https://delta-diffusion.github.io/.
A new video generation method, $\delta$-Diffusion, learns from unlabeled videos to produce realistic continuations based on demonstrations, using implicit latent control and a video foundation model with appearance bottlenecks.
video creation by demonstration $\delta$-Diffusion self-supervised training conditional future frame prediction implicit latent control video foundation model appearance bottleneck action latents human preference large-scale machine evaluations interactive world simulation