Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video
  Generation
Significant advancements have been achieved in the realm of large-scale
pre-trained text-to-video Diffusion Models (VDMs). However, previous methods
either rely solely on pixel-based VDMs, which come with high computational
costs, or on latent-based VDMs, which often struggle with precise text-video
alignment. In this paper, we are the first to propose a hybrid model, dubbed as
Show-1, which marries pixel-based and latent-based VDMs for text-to-video
generation. Our model first uses pixel-based VDMs to produce a low-resolution
video of strong text-video correlation. After that, we propose a novel expert
translation method that employs the latent-based VDMs to further upsample the
low-resolution video to high resolution. Compared to latent VDMs, Show-1 can
produce high-quality videos of precise text-video alignment; Compared to pixel
VDMs, Show-1 is much more efficient (GPU memory usage during inference is 15G
vs 72G). We also validate our model on standard video generation benchmarks.
Our code and model weights are publicly available at
https://github.com/showlab/Show-1.
651507604a40f143596de2f2
David Junhao Zhang
admin_assigned
2023-09-28T08:57:39.392Z
6468855763a564ba347e1efa
/avatars/caa99c3318c1d6d21be44d2d32795a62.svg
David Junhao ZHANG
user
Junhao233
651507604a40f143596de2f3
Jay Zhangjie Wu
claimed_verified
2023-09-28T08:20:16.342Z
633aaf695df91da9cea92960
https://cdn-avatars.huggingface.co/v1/production/uploads/633aaf695df91da9cea92960/h1oOWUudbdlsd63Q5w0hM.png
Jay Wu
user
jayw
651507604a40f143596de2f4
Jia-Wei Liu
admin_assigned
2023-09-28T09:04:35.169Z
64230048a73327caad9d8241
/avatars/ac115f1a21c743db7c925f0e18451145.svg
Jiawei Liu
user
JiaweiLIU
651507604a40f143596de2f5
Rui Zhao
claimed_verified
2023-10-16T07:25:07.548Z
652b83b73b5997ed71a310f2
https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
Rui Zhao
user
ruizhaocv
651507604a40f143596de2f6
Lingmin Ran
651507604a40f143596de2f7
Yuchao Gu
651507604a40f143596de2f8
Difei Gao
651507604a40f143596de2f9
Mike Zheng Shou
claimed_verified
2023-09-28T09:01:32.912Z
63a55320ce5763e06f78519c
https://cdn-avatars.huggingface.co/v1/production/uploads/1671779060549-noauth.jpeg
Mike Shou
user
mikeshou
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video
  Generation
Significant advancements have been achieved in the realm of large-scale
pre-trained text-to-video Diffusion Models (VDMs). However, previous methods
either rely solely on pixel-based VDMs, which come with high computational
costs, or on latent-based VDMs, which often struggle with precise text-video
alignment. In this paper, we are the first to propose a hybrid model, dubbed as
Show-1, which marries pixel-based and latent-based VDMs for text-to-video
generation. Our model first uses pixel-based VDMs to produce a low-resolution
video of strong text-video correlation. After that, we propose a novel expert
translation method that employs the latent-based VDMs to further upsample the
low-resolution video to high resolution. Compared to latent VDMs, Show-1 can
produce high-quality videos of precise text-video alignment; Compared to pixel
VDMs, Show-1 is much more efficient (GPU memory usage during inference is 15G
vs 72G). We also validate our model on standard video generation benchmarks.
Our code and model weights are publicly available at
https://github.com/showlab/Show-1.
Show-1, a hybrid model combining pixel-based and latent-based VDMs, generates high-quality text-to-video with precise alignment and reduced computational cost.
Diffusion Models (VDMs) pixel-based VDMs latent-based VDMs text-video correlation expert translation method text-video alignment GPU memory usage