LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion
  Models
This work aims to learn a high-quality text-to-video (T2V) generative model
by leveraging a pre-trained text-to-image (T2I) model as a basis. It is a
highly desirable yet challenging task to simultaneously a) accomplish the
synthesis of visually realistic and temporally coherent videos while b)
preserving the strong creative generation nature of the pre-trained T2I model.
To this end, we propose LaVie, an integrated video generation framework that
operates on cascaded video latent diffusion models, comprising a base T2V
model, a temporal interpolation model, and a video super-resolution model. Our
key insights are two-fold: 1) We reveal that the incorporation of simple
temporal self-attentions, coupled with rotary positional encoding, adequately
captures the temporal correlations inherent in video data. 2) Additionally, we
validate that the process of joint image-video fine-tuning plays a pivotal role
in producing high-quality and creative outcomes. To enhance the performance of
LaVie, we contribute a comprehensive and diverse video dataset named Vimeo25M,
consisting of 25 million text-video pairs that prioritize quality, diversity,
and aesthetic appeal. Extensive experiments demonstrate that LaVie achieves
state-of-the-art performance both quantitatively and qualitatively.
Furthermore, we showcase the versatility of pre-trained LaVie models in various
long video generation and personalized video synthesis applications.
651385b6ac120f23e0053904
Yaohui Wang
admin_assigned
2023-09-27T10:03:07.095Z
63201256c6b20f03c829c4b8
/avatars/a42092119777d65e60b12eb5ba0e45f1.svg
Yaohui Wang
user
YaohuiW
651385b6ac120f23e0053905
Xinyuan Chen
admin_assigned
2023-09-27T10:03:16.425Z
643e943a70c6a27621eb1c89
/avatars/73ec521ab5ba84cc7908c52c0acef6ef.svg
Xinyuan Chen
user
AriaChen
651385b6ac120f23e0053906
Xin Ma
claimed_verified
2023-09-28T08:20:25.748Z
64a51faf9ec0167a99a4d114
https://cdn-avatars.huggingface.co/v1/production/uploads/64a51faf9ec0167a99a4d114/IlJS0mwkF_ZIebaQrbJIl.jpeg
Xin Ma
user
maxin-cn
651385b6ac120f23e0053907
Shangchen Zhou
admin_assigned
2023-09-27T10:04:30.137Z
62e57662ae9d3f10acbb1b1b
https://cdn-avatars.huggingface.co/v1/production/uploads/62e57662ae9d3f10acbb1b1b/lg58jdbNyv6LGH2LFnZDF.png
Shangchen Zhou
user
sczhou
651385b6ac120f23e0053908
Ziqi Huang
claimed_verified
2023-09-27T07:26:53.301Z
60efe7fa0d920bc7805cada5
https://cdn-avatars.huggingface.co/v1/production/uploads/60efe7fa0d920bc7805cada5/2LBrJBjSCOP5ilZIpWLHl.png
Ziqi Huang
user
Ziqi
651385b6ac120f23e0053909
Yi Wang
651385b6ac120f23e005390a
Ceyuan Yang
admin_assigned
2023-09-27T10:05:59.235Z
64accf3fc1da7c4dbc69392d
/avatars/ec8ad2498f8a49a7a00c77a7f70d34bb.svg
Ceyuan
user
Ceyuan
651385b6ac120f23e005390b
Yinan He
claimed_verified
2023-10-09T07:53:14.524Z
62aafa49f29ff279b51f0182
https://cdn-avatars.huggingface.co/v1/production/uploads/62aafa49f29ff279b51f0182/rQx8QFQGOY2qIhqJ8zSRj.jpeg
yinanhe
user
ynhe
651385b6ac120f23e005390c
Jiashuo Yu
admin_assigned
2023-09-27T10:06:21.215Z
64b89a14cf14c2fabe96664c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tEd3fBjMcEubF4plqzcUz.jpeg
Jiashuo Yu
user
awojustin
651385b6ac120f23e005390d
Peiqing Yang
admin_assigned
2023-09-27T10:06:28.910Z
6513aae6330c55fdc5462ca8
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EDhpiTqCBMNPmMGrOKcvY.jpeg
pq-yang
user
PeiqingYang
651385b6ac120f23e005390e
Yuwei Guo
admin_assigned
2023-09-27T10:08:47.559Z
6371f83d5ffa922d638ef486
https://cdn-avatars.huggingface.co/v1/production/uploads/6371f83d5ffa922d638ef486/UzWl3xhVvtol3Rz1mjQ9t.jpeg
Yuwei Guo
user
guoyww
651385b6ac120f23e005390f
Tianxing Wu
admin_assigned
2023-09-27T10:08:56.931Z
63185a8973e9e4aa51dbc0ce
https://cdn-avatars.huggingface.co/v1/production/uploads/1676359176854-63185a8973e9e4aa51dbc0ce.jpeg
Tianxing Wu
user
TianxingWu
651385b6ac120f23e0053910
Chenyang Si
admin_assigned
2023-09-27T10:09:04.159Z
635f8ed47c05eb9f59963d3a
https://cdn-avatars.huggingface.co/v1/production/uploads/635f8ed47c05eb9f59963d3a/uQf4p9N9pSaFy87Wg9v4k.jpeg
ChenyangSi
user
ChenyangSi
651385b6ac120f23e0053911
Yuming Jiang
admin_assigned
2023-09-27T10:09:11.257Z
629c95b7a5d6f5fe10e6ed45
https://cdn-avatars.huggingface.co/v1/production/uploads/629c95b7a5d6f5fe10e6ed45/Sy0Ype5snsRookID-gsSm.jpeg
Yuming Jiang
user
yumingj
651385b6ac120f23e0053912
Cunjian Chen
651385b6ac120f23e0053913
Chen Change Loy
651385b6ac120f23e0053914
Bo Dai
admin_assigned
2023-09-27T10:09:34.383Z
635f93577c05eb9f59966209
/avatars/add48d7e3790a6b500d6c451ef8b0f75.svg
Intelligent Digital Creation
user
BoDai
651385b6ac120f23e0053915
Dahua Lin
admin_assigned
2023-09-27T10:09:42.739Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
651385b6ac120f23e0053916
Yu Qiao
651385b6ac120f23e0053917
Ziwei Liu
admin_assigned
2023-09-27T10:10:55.550Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion
  Models
This work aims to learn a high-quality text-to-video (T2V) generative model
by leveraging a pre-trained text-to-image (T2I) model as a basis. It is a
highly desirable yet challenging task to simultaneously a) accomplish the
synthesis of visually realistic and temporally coherent videos while b)
preserving the strong creative generation nature of the pre-trained T2I model.
To this end, we propose LaVie, an integrated video generation framework that
operates on cascaded video latent diffusion models, comprising a base T2V
model, a temporal interpolation model, and a video super-resolution model. Our
key insights are two-fold: 1) We reveal that the incorporation of simple
temporal self-attentions, coupled with rotary positional encoding, adequately
captures the temporal correlations inherent in video data. 2) Additionally, we
validate that the process of joint image-video fine-tuning plays a pivotal role
in producing high-quality and creative outcomes. To enhance the performance of
LaVie, we contribute a comprehensive and diverse video dataset named Vimeo25M,
consisting of 25 million text-video pairs that prioritize quality, diversity,
and aesthetic appeal. Extensive experiments demonstrate that LaVie achieves
state-of-the-art performance both quantitatively and qualitatively.
Furthermore, we showcase the versatility of pre-trained LaVie models in various
long video generation and personalized video synthesis applications.
LaVie, a text-to-video framework integrating cascaded video latent diffusion models, achieves state-of-the-art video generation by leveraging temporal self-attentions and joint image-video fine-tuning.
text-to-video text-to-image latent diffusion models temporal self-attentions rotary positional encoding video super-resolution joint image-video fine-tuning Vimeo25M