Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models
Despite tremendous progress in generating high-quality images using diffusion
models, synthesizing a sequence of animated frames that are both photorealistic
and temporally coherent is still in its infancy. While off-the-shelf
billion-scale datasets for image generation are available, collecting similar
video data of the same scale is still challenging. Also, training a video
diffusion model is computationally much more expensive than its image
counterpart. In this work, we explore finetuning a pretrained image diffusion
model with video data as a practical solution for the video synthesis task. We
find that naively extending the image noise prior to video noise prior in video
diffusion leads to sub-optimal performance. Our carefully designed video noise
prior leads to substantially better performance. Extensive experimental
validation shows that our model, Preserve Your Own Correlation (PYoCo), attains
SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks. It
also achieves SOTA video generation quality on the small-scale UCF-101
benchmark with a 10times smaller model using significantly less computation
than the prior art.
6466d620060756d2854abb76
Songwei Ge
claimed_verified
2023-05-23T08:11:29.465Z
631cdc696e623740a78fd864
/avatars/f610873f6080453b655fb230d6f3b765.svg
Songwei Ge
user
songweig
6466d620060756d2854abb77
Seungjun Nah
admin_assigned
2023-05-21T20:34:11.354Z
62f6d3278465979f627e00ff
/avatars/4b96c355e6d8e9db0a23eaf85da7b215.svg
Seungjun Nah
user
snah
6466d620060756d2854abb78
Guilin Liu
claimed_verified
2025-04-22T10:18:43.129Z
63bc4115141c7d395c4aa02c
/avatars/a9eca59e8c52cceab6524a4aaab11e38.svg
Liu
user
liuguilin
6466d620060756d2854abb79
Tyler Poon
6466d620060756d2854abb7a
Andrew Tao
6466d620060756d2854abb7b
Bryan Catanzaro
admin_assigned
2023-05-21T20:35:39.255Z
6311021788942700629e6247
/avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
Bryan Catanzaro
user
ctnzr
6466d620060756d2854abb7c
David Jacobs
6466d620060756d2854abb7d
Jia-Bin Huang
6466d620060756d2854abb7e
Ming-Yu Liu
admin_assigned
2023-05-21T20:36:52.721Z
62f049afdf4b93aad5c7f2d6
/avatars/e272e58ad996733d7098e50248e5b57e.svg
Ming-Yu Liu
user
mingyuliutw
6466d620060756d2854abb7f
Yogesh Balaji
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models
Despite tremendous progress in generating high-quality images using diffusion
models, synthesizing a sequence of animated frames that are both photorealistic
and temporally coherent is still in its infancy. While off-the-shelf
billion-scale datasets for image generation are available, collecting similar
video data of the same scale is still challenging. Also, training a video
diffusion model is computationally much more expensive than its image
counterpart. In this work, we explore finetuning a pretrained image diffusion
model with video data as a practical solution for the video synthesis task. We
find that naively extending the image noise prior to video noise prior in video
diffusion leads to sub-optimal performance. Our carefully designed video noise
prior leads to substantially better performance. Extensive experimental
validation shows that our model, Preserve Your Own Correlation (PYoCo), attains
SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks. It
also achieves SOTA video generation quality on the small-scale UCF-101
benchmark with a 10times smaller model using significantly less computation
than the prior art.
A pretrained image diffusion model fine-tuned with video data achieves state-of-the-art video synthesis quality and zero-shot text-to-video results with reduced computational cost.
diffusion models video noise prior image noise prior zero-shot text-to-video video generation quality UCF-101 MSR-VTT model fine-tuning