Lumiere: A Space-Time Diffusion Model for Video Generation
We introduce Lumiere -- a text-to-video diffusion model designed for
synthesizing videos that portray realistic, diverse and coherent motion -- a
pivotal challenge in video synthesis. To this end, we introduce a Space-Time
U-Net architecture that generates the entire temporal duration of the video at
once, through a single pass in the model. This is in contrast to existing video
models which synthesize distant keyframes followed by temporal super-resolution
-- an approach that inherently makes global temporal consistency difficult to
achieve. By deploying both spatial and (importantly) temporal down- and
up-sampling and leveraging a pre-trained text-to-image diffusion model, our
model learns to directly generate a full-frame-rate, low-resolution video by
processing it in multiple space-time scales. We demonstrate state-of-the-art
text-to-video generation results, and show that our design easily facilitates a
wide range of content creation tasks and video editing applications, including
image-to-video, video inpainting, and stylized generation.
65b06dc7fdf08903343bfeae
Omer Bar-Tal
admin_assigned
2024-01-24T09:09:25.587Z
62e29044a133a252b5cf70b2
/avatars/6d09ddcba9bc47c309150a8d77815891.svg
Omer Bar-Tal
user
omerbartal
65b06dc7fdf08903343bfeaf
Hila Chefer
claimed_verified
2024-01-24T08:08:47.043Z
6181c72cdcc1df2c9de8a4d8
https://cdn-avatars.huggingface.co/v1/production/uploads/1655248010394-6181c72cdcc1df2c9de8a4d8.jpeg
Hila Chefer
user
Hila
65b06dc7fdf08903343bfeb0
Omer Tov
admin_assigned
2024-01-24T09:09:32.274Z
623889093657dc674c30954e
/avatars/a5a193d231e01448cc78ed5bba52bff5.svg
Omer Tov
user
omertov
65b06dc7fdf08903343bfeb1
Charles Herrmann
admin_assigned
2024-01-24T09:09:39.014Z
64c7f088c864d962eddeaf8c
/avatars/e6364c56e6cb6907b6dd2d34eed83720.svg
Charles Herrmann
user
irwinherrmann
65b06dc7fdf08903343bfeb2
Roni Paiss
admin_assigned
2024-01-24T09:09:45.323Z
62cab8aeb189b6164d26bfbe
/avatars/7ed117e375dd9280a5750a242cae24ca.svg
Roni Paiss
user
RoniPaiss
65b06dc7fdf08903343bfeb3
Shiran Zada
admin_assigned
2024-01-24T09:09:51.755Z
63510f62cba4ff2e81cb0492
/avatars/2375a841491de8f40f66b6d0fb0df7b1.svg
Shiran Zada
user
shiranzada
65b06dc7fdf08903343bfeb4
Ariel Ephrat
claimed_verified
2024-01-29T08:26:18.807Z
65b6ada5ff5235e6ca9551c9
/avatars/2c558fd133bc1d0cd1b5d7f39d1968ee.svg
Ariel Ephrat
user
arielephrat
65b06dc7fdf08903343bfeb5
Junhwa Hur
admin_assigned
2024-01-24T09:10:02.609Z
647f3046b0e96764589e107a
/avatars/1074ee5ec1b2d5d297406f9d455213c4.svg
Junhwa Hur
user
hurjunhwa
65b06dc7fdf08903343bfeb6
Yuanzhen Li
admin_assigned
2024-01-24T09:10:09.523Z
6334ee5831a2be3938c85639
/avatars/9821af26a585f54b595ec39f061f2250.svg
Yuanzhen Li
user
yzli
65b06dc7fdf08903343bfeb7
Tomer Michaeli
admin_assigned
2024-01-24T09:10:16.729Z
6457c3f50664b05a5b1d941d
/avatars/0132be4d068d4b314f6b498b65411461.svg
Tomer Michaeli
user
Michato
65b06dc7fdf08903343bfeb8
Oliver Wang
admin_assigned
2024-01-24T09:11:23.096Z
62fa8bfdc95d426ff8f25fe9
/avatars/71fe9cff4ce79f6a208045984852e5ad.svg
Oliver Wang
user
oliverwang
65b06dc7fdf08903343bfeb9
Deqing Sun
65b06dc7fdf08903343bfeba
Tali Dekel
admin_assigned
2024-01-24T09:11:37.971Z
631cddec68f7da9ad24f6fc7
/avatars/7d4f1ce805e5889ca6594bd4a93f2583.svg
Tali Dekel
user
talidekel
65b06dc7fdf08903343bfebb
Inbar Mosseri
Lumiere: A Space-Time Diffusion Model for Video Generation
We introduce Lumiere -- a text-to-video diffusion model designed for
synthesizing videos that portray realistic, diverse and coherent motion -- a
pivotal challenge in video synthesis. To this end, we introduce a Space-Time
U-Net architecture that generates the entire temporal duration of the video at
once, through a single pass in the model. This is in contrast to existing video
models which synthesize distant keyframes followed by temporal super-resolution
-- an approach that inherently makes global temporal consistency difficult to
achieve. By deploying both spatial and (importantly) temporal down- and
up-sampling and leveraging a pre-trained text-to-image diffusion model, our
model learns to directly generate a full-frame-rate, low-resolution video by
processing it in multiple space-time scales. We demonstrate state-of-the-art
text-to-video generation results, and show that our design easily facilitates a
wide range of content creation tasks and video editing applications, including
image-to-video, video inpainting, and stylized generation.
A text-to-video diffusion model using Space-Time U-Net architecture generates realistic, diverse, and coherent videos through a single pass, achieving state-of-the-art results and supporting various content creation and editing tasks.
Space-Time U-Net diffusion model temporal duration spatial down- and up-sampling temporal down- and up-sampling text-to-image diffusion model full-frame-rate low-resolution video image-to-video video inpainting stylized generation