Emu Video: Factorizing Text-to-Video Generation by Explicit Image
  Conditioning
We present Emu Video, a text-to-video generation model that factorizes the
generation into two steps: first generating an image conditioned on the text,
and then generating a video conditioned on the text and the generated image. We
identify critical design decisions--adjusted noise schedules for diffusion, and
multi-stage training--that enable us to directly generate high quality and high
resolution videos, without requiring a deep cascade of models as in prior work.
In human evaluations, our generated videos are strongly preferred in quality
compared to all prior work--81% vs. Google's Imagen Video, 90% vs. Nvidia's
PYOCO, and 96% vs. Meta's Make-A-Video. Our model outperforms commercial
solutions such as RunwayML's Gen2 and Pika Labs. Finally, our factorizing
approach naturally lends itself to animating images based on a user's text
prompt, where our generations are preferred 96% over prior work.
655ac402deee83130a46b65b
Rohit Girdhar
admin_assigned
2023-11-20T10:09:09.220Z
63c6c737656e7822e23732d8
/avatars/3cf062864dbf2b5423eb1f4beaeec1ea.svg
Rohit Girdhar
user
rgirdhar
655ac402deee83130a46b65c
Mannat Singh
claimed_verified
2023-11-23T10:49:19.549Z
650fece35877b1c077298b81
/avatars/be9c948ecf9c413ad4c8aa6598d3ba86.svg
Mannat Singh
user
mannatsingh
655ac402deee83130a46b65d
Andrew Brown
655ac402deee83130a46b65e
Quentin Duval
655ac402deee83130a46b65f
Samaneh Azadi
655ac402deee83130a46b660
Sai Saketh Rambhatla
admin_assigned
2023-11-20T10:10:22.412Z
6435a89b876d46610bf30b21
/avatars/88dba6589b07ad310bad94108774d893.svg
Saketh Rambhatla
user
rssaketh
655ac402deee83130a46b661
Akbar Shah
655ac402deee83130a46b662
Xi Yin
655ac402deee83130a46b663
Devi Parikh
admin_assigned
2023-11-20T10:10:50.917Z
6340753995c20b9447379c67
/avatars/f793c00a98335cd8f7efce41d403da16.svg
Devi Parikh
user
deviparikh-genai
655ac402deee83130a46b664
Ishan Misra
Emu Video: Factorizing Text-to-Video Generation by Explicit Image
  Conditioning
We present Emu Video, a text-to-video generation model that factorizes the
generation into two steps: first generating an image conditioned on the text,
and then generating a video conditioned on the text and the generated image. We
identify critical design decisions--adjusted noise schedules for diffusion, and
multi-stage training--that enable us to directly generate high quality and high
resolution videos, without requiring a deep cascade of models as in prior work.
In human evaluations, our generated videos are strongly preferred in quality
compared to all prior work--81% vs. Google's Imagen Video, 90% vs. Nvidia's
PYOCO, and 96% vs. Meta's Make-A-Video. Our model outperforms commercial
solutions such as RunwayML's Gen2 and Pika Labs. Finally, our factorizing
approach naturally lends itself to animating images based on a user's text
prompt, where our generations are preferred 96% over prior work.
A text-to-video generation model that factors video creation into two steps—image generation followed by video generation—achieves superior quality and resolution without deep model cascades.
text-to-video generation diffusion noise schedules multi-stage training image generation video generation human evaluations Imagen Video PYOCO Make-A-Video Gen2 Pika Labs