Photorealistic Video Generation with Diffusion Models
We present W.A.L.T, a transformer-based approach for photorealistic video
generation via diffusion modeling. Our approach has two key design decisions.
First, we use a causal encoder to jointly compress images and videos within a
unified latent space, enabling training and generation across modalities.
Second, for memory and training efficiency, we use a window attention
architecture tailored for joint spatial and spatiotemporal generative modeling.
Taken together these design decisions enable us to achieve state-of-the-art
performance on established video (UCF-101 and Kinetics-600) and image
(ImageNet) generation benchmarks without using classifier free guidance.
Finally, we also train a cascade of three models for the task of text-to-video
generation consisting of a base latent video diffusion model, and two video
super-resolution diffusion models to generate videos of 512 times 896
resolution at 8 frames per second.
6577f3b058d7a2cc89342628
Agrim Gupta
admin_assigned
2023-12-12T08:17:28.178Z
63defb9015266dd945f9f772
/avatars/b304ac6c3b45553358f8b6cd5c3df293.svg
Agrim
user
agrimgupta
6577f3b058d7a2cc89342629
Lijun Yu
admin_assigned
2023-12-12T08:17:49.551Z
639886ba65a34085ae29abb7
/avatars/51f4d4d6fbcc07ab8b7a4c1c2f7da53d.svg
Lijun Yu
user
lijunyu
6577f3b058d7a2cc8934262a
Kihyuk Sohn
6577f3b058d7a2cc8934262b
Xiuye Gu
6577f3b058d7a2cc8934262c
Meera Hahn
claimed_verified
2025-07-29T07:14:00.669Z
6887d4b7d01ee133939109b9
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/bCUDzOvi2FFQVzOzmGBaW.png
Meera Hahn
user
meerahahn
6577f3b058d7a2cc8934262d
Li Fei-Fei
6577f3b058d7a2cc8934262e
Irfan Essa
6577f3b058d7a2cc8934262f
Lu Jiang
claimed_verified
2023-12-23T11:18:10.788Z
64a5cba3bea0116f8f7187a7
/avatars/97b3bad82e359c57d1ab7d6e3aa7748b.svg
Lu Jiang
user
roadjiang
6577f3b058d7a2cc89342630
José Lezama
Photorealistic Video Generation with Diffusion Models
We present W.A.L.T, a transformer-based approach for photorealistic video
generation via diffusion modeling. Our approach has two key design decisions.
First, we use a causal encoder to jointly compress images and videos within a
unified latent space, enabling training and generation across modalities.
Second, for memory and training efficiency, we use a window attention
architecture tailored for joint spatial and spatiotemporal generative modeling.
Taken together these design decisions enable us to achieve state-of-the-art
performance on established video (UCF-101 and Kinetics-600) and image
(ImageNet) generation benchmarks without using classifier free guidance.
Finally, we also train a cascade of three models for the task of text-to-video
generation consisting of a base latent video diffusion model, and two video
super-resolution diffusion models to generate videos of 512 times 896
resolution at 8 frames per second.
A transformer-based diffusion model using causal encoding and window attention generates high-resolution, photorealistic videos, achieving state-of-the-art performance without classifier-free guidance and includes a cascade for text-to-video generation.
transformer-based approach diffusion modeling causal encoder unified latent space joint spatial spatiotemporal generative modeling window attention architecture video generation ImageNet UCF-101 Kinetics-600 latent video diffusion model video super-resolution diffusion models text-to-video generation