Movie Gen: A Cast of Media Foundation Models
We present Movie Gen, a cast of foundation models that generates
high-quality, 1080p HD videos with different aspect ratios and synchronized
audio. We also show additional capabilities such as precise instruction-based
video editing and generation of personalized videos based on a user's image.
Our models set a new state-of-the-art on multiple tasks: text-to-video
synthesis, video personalization, video editing, video-to-audio generation, and
text-to-audio generation. Our largest video generation model is a 30B parameter
transformer trained with a maximum context length of 73K video tokens,
corresponding to a generated video of 16 seconds at 16 frames-per-second. We
show multiple technical innovations and simplifications on the architecture,
latent spaces, training objectives and recipes, data curation, evaluation
protocols, parallelization techniques, and inference optimizations that allow
us to reap the benefits of scaling pre-training data, model size, and training
compute for training large scale media generation models. We hope this paper
helps the research community to accelerate progress and innovation in media
generation models. All videos from this paper are available at
https://go.fb.me/MovieGenResearchVideos.
6711d977b265bbd413b8320c
Adam Polyak
admin_assigned
2024-10-18T12:16:24.972Z
6304d514dae2eb7d08413d62
/avatars/02a571bc791b78d3993d9a0484b70a29.svg
Adam Polyak
user
adampo
6711d977b265bbd413b8320d
Amit Zohar
admin_assigned
2024-10-18T12:16:30.835Z
64d8cb34505306fcd2fb89c3
/avatars/4ae9ffe5d494a7d53bc519ff8e402b3e.svg
Amit Zohar
user
amitz
6711d977b265bbd413b8320e
Andrew Brown
6711d977b265bbd413b8320f
Andros Tjandra
admin_assigned
2024-10-18T12:16:39.991Z
647779c6c21496284db45835
/avatars/c1176e57f5d0aaa5f50bbde48a19af01.svg
Andros Tjandra
user
androstj
6711d977b265bbd413b83210
Animesh Sinha
admin_assigned
2024-10-18T12:17:19.207Z
63772d28e069bdcdbb8e4a29
/avatars/be71b00e009f2093a3ff79f07815e68e.svg
Animesh Sinha
user
animeshsinha
6711d977b265bbd413b83211
Ann Lee
admin_assigned
2024-10-18T12:17:31.419Z
663353017263322326bb044e
/avatars/ac6a5b814592e4c3cd84d7255e31ec88.svg
Yi-hsuan Lee
user
AnnLee
6711d977b265bbd413b83212
Apoorv Vyas
admin_assigned
2024-10-18T12:17:01.338Z
638697308b5acae8d251cdec
/avatars/dcfc47ee012439cc5ecc28e48ff26983.svg
Apoorv Vyas
user
apoorv2904
6711d977b265bbd413b83213
Bowen Shi
admin_assigned
2024-10-18T12:17:46.422Z
6477d6e06e6c7ac608c44734
/avatars/c6d126c5b2930531ac4b00a1ad5e6d26.svg
Bowen Shi
user
bowenshi
6711d977b265bbd413b83214
Chih-Yao Ma
admin_assigned
2024-10-18T12:18:03.265Z
6515bf00c2aedaa7e692a1ec
https://cdn-avatars.huggingface.co/v1/production/uploads/6515bf00c2aedaa7e692a1ec/PYpS1QwTPdQYW5_BrvwzP.png
Chih-Yao Ma
user
chihyaoma
6711d977b265bbd413b83215
Ching-Yao Chuang
6711d977b265bbd413b83216
David Yan
6711d977b265bbd413b83217
Dhruv Choudhary
6711d977b265bbd413b83218
Dingkang Wang
admin_assigned
2024-10-18T12:19:33.523Z
65dbd00d730386b492a88c06
/avatars/362ae2bf1144d0bbb822fad3c33f239c.svg
Dingkang Wang
user
wangdk93
6711d977b265bbd413b83219
Geet Sethi
admin_assigned
2024-10-18T12:19:43.218Z
653405a002d1ecd5455d4f0a
/avatars/fc5ab33a508dadec6d2e15e992e16dec.svg
Geet Sethi
user
geetsethi
6711d977b265bbd413b8321a
Guan Pang
admin_assigned
2024-10-18T12:19:51.272Z
6573a458d7f487de5f9bb44a
/avatars/4aa5626af43855ad770186f0d4191a01.svg
Guan Pang
user
tbpangolin
6711d977b265bbd413b8321b
Haoyu Ma
admin_assigned
2024-10-18T12:20:03.497Z
650a8979c19e5b4c8a6ff062
https://cdn-avatars.huggingface.co/v1/production/uploads/650a8979c19e5b4c8a6ff062/64_JuECX_k_-uK7m7nlua.jpeg
Haoyu Ma
user
haoyum1997
6711d977b265bbd413b8321c
Ishan Misra
admin_assigned
2024-10-18T12:20:11.699Z
63faa5b04380ab0cb9577a57
/avatars/c71be0acb0553671c9703654e998cbd5.svg
Ishan Misra
user
hayhay404
6711d977b265bbd413b8321d
Ji Hou
claimed_verified
2025-04-13T19:28:14.800Z
63084636f0dc38fb47c2dc6b
/avatars/04edff10a048b5d93981e6f5af3ecf40.svg
Ji Hou
user
jhou90
6711d977b265bbd413b8321e
Jialiang Wang
6711d977b265bbd413b8321f
Kiran Jagadeesh
6711d977b265bbd413b83220
Kunpeng Li
6711d977b265bbd413b83221
Luxin Zhang
claimed_verified
2025-04-02T08:56:26.554Z
67ecad336560da48c5c2bf08
/avatars/5ef8381504bfc226b0d82c5a36f04ca3.svg
Luxin Zhang
user
luxinzhang
6711d977b265bbd413b83222
Mannat Singh
admin_assigned
2024-10-18T12:22:01.617Z
650fece35877b1c077298b81
/avatars/be9c948ecf9c413ad4c8aa6598d3ba86.svg
Mannat Singh
user
mannatsingh
6711d977b265bbd413b83223
Mary Williamson
6711d977b265bbd413b83224
Matt Le
claimed_verified
2024-10-18T14:16:58.580Z
6311fba9fdb55de45d2245ea
/avatars/5799e56b737d78550bf8d164cbe3d41c.svg
Matt Le
user
lematt1991
6711d977b265bbd413b83225
Matthew Yu
6711d977b265bbd413b83226
Mitesh Kumar Singh
6711d977b265bbd413b83227
Peizhao Zhang
admin_assigned
2024-10-18T12:22:59.119Z
6433a635a5aed21dd110381e
/avatars/b1fa4fe1c8bd7989875233820fa425af.svg
Peizhao Zhang
user
stzpz
6711d977b265bbd413b83228
Peter Vajda
admin_assigned
2024-10-18T12:23:12.630Z
636f4b35af79dd193603342b
/avatars/577328195738d92ac41a3d0ff3108d7c.svg
Peter Vajda
user
vaphab
6711d977b265bbd413b83229
Quentin Duval
admin_assigned
2024-10-18T12:23:18.989Z
66a1632f40c32a6a39ba0c51
/avatars/5f577e016fa68e65b72fb1f154c88288.svg
Quentin Duval
user
qduval
6711d977b265bbd413b8322a
Rohit Girdhar
admin_assigned
2024-10-18T12:23:27.698Z
63c6c737656e7822e23732d8
/avatars/3cf062864dbf2b5423eb1f4beaeec1ea.svg
Rohit Girdhar
user
rgirdhar
6711d977b265bbd413b8322b
Roshan Sumbaly
admin_assigned
2024-10-18T12:23:34.133Z
6515c595f464644a73b50064
/avatars/119102c41fb8a7986c72b3f55ec6200f.svg
Roshan sumbaly
user
rsumbaly
6711d977b265bbd413b8322c
Sai Saketh Rambhatla
6711d977b265bbd413b8322d
Sam Tsai
claimed_verified
2024-10-18T15:31:35.305Z
67127b753a058517519fabe8
/avatars/6fabb1634efb8ad8a8a2ae96fd6cf1c6.svg
Sam Tsai
user
sstsai-adl
6711d977b265bbd413b8322e
Samaneh Azadi
6711d977b265bbd413b8322f
Samyak Datta
admin_assigned
2024-10-18T12:24:02.672Z
657c9c9242fc53e18b00a3bc
/avatars/0365d677a8151632f05eff140d8cdb62.svg
Samyak Datta
user
samyakdatta
6711d977b265bbd413b83230
Sanyuan Chen
admin_assigned
2024-10-18T12:24:10.750Z
6465843ce27766e8921b9cab
/avatars/6017db24da593c9afb21edfcedcddfd4.svg
Sanyuan Chen
user
Sanyuan-Chen
6711d977b265bbd413b83231
Sean Bell
6711d977b265bbd413b83232
Sharadh Ramaswamy
6711d977b265bbd413b83233
Shelly Sheynin
admin_assigned
2024-10-18T12:24:59.545Z
63468e6ca6f101c7f9049132
https://cdn-avatars.huggingface.co/v1/production/uploads/1665662111525-63468e6ca6f101c7f9049132.png
Shelly Sheynin
user
shellysheynin
6711d977b265bbd413b83234
Siddharth Bhattacharya
admin_assigned
2024-10-18T12:24:19.232Z
65d3c3fe974775cbcbf3397a
/avatars/06c1a26cee0ffad7d318a24c4a16ad9d.svg
Siddharth Bhattacharya
user
Siddharth96
6711d977b265bbd413b83235
Simran Motwani
admin_assigned
2024-10-18T12:25:07.110Z
6515bb7e99e8060f1e792047
/avatars/9c6400f83c3165de3154b0e6bee2e5a1.svg
Simran Motwani
user
smotwani
6711d977b265bbd413b83236
Tao Xu
admin_assigned
2024-10-18T12:25:15.774Z
632bc84fca316c73cd40ca5c
/avatars/5b000f86f8161f7a4bc4200e4b76a6e1.svg
Tao Xu
user
taoxu
6711d977b265bbd413b83237
Tianhe Li
6711d977b265bbd413b83238
Tingbo Hou
admin_assigned
2024-10-18T12:25:44.779Z
655846d7ed8df83128f5826a
/avatars/d7ce174d7d1b8614d5f6f071225c0057.svg
Hou
user
Tingbo
6711d977b265bbd413b83239
Wei-Ning Hsu
admin_assigned
2024-10-18T12:25:56.603Z
64c143d484887aa2b43bac0d
/avatars/817dba24739676aa0f34692702bb00d6.svg
Wei-Ning Hsu
user
wnhsu
6711d977b265bbd413b8323a
Xi Yin
6711d977b265bbd413b8323b
Xiaoliang Dai
admin_assigned
2024-10-18T12:26:10.362Z
6549417b3ce45eb764faf993
/avatars/d310f475d0697f5f13b3d4141ea0ccaf.svg
Xiaoliang Dai
user
daixl1992
6711d977b265bbd413b8323c
Yaniv Taigman
6711d977b265bbd413b8323d
Yaqiao Luo
admin_assigned
2024-10-18T12:26:55.559Z
63d2a5a6c20657d24347000c
https://cdn-avatars.huggingface.co/v1/production/uploads/1674749322268-noauth.jpeg
Luo
user
Yaqiao
6711d977b265bbd413b8323e
Yen-Cheng Liu
admin_assigned
2024-10-18T12:27:11.636Z
620607bc980cf06cf5b1cd3b
/avatars/3a49b4bd068ca4ec9e16edd65862ce5b.svg
Yen-Cheng Liu
user
ycliu
6711d977b265bbd413b8323f
Yi-Chiao Wu
admin_assigned
2024-10-18T12:27:04.005Z
6410ebe8028e6f7153df0842
/avatars/96960d2c7d713510d25903fdabe5b825.svg
Yi-Chiao Wu
user
yichiaowu
6711d977b265bbd413b83240
Yue Zhao
claimed_verified
2024-10-25T09:31:09.407Z
641cc612f0b71a9743668604
/avatars/ab7d91df8de266fadb4e5632a1f0a37a.svg
Yue (R) Zhao
user
yzhao30
6711d977b265bbd413b83241
Yuval Kirstain
admin_assigned
2024-10-18T12:27:22.706Z
6551fa1b5491c57ee900108b
https://cdn-avatars.huggingface.co/v1/production/uploads/6551fa1b5491c57ee900108b/HXEfGUJRExQpfGpX3F8N0.png
Yuval Kirstain
user
yuvalkirstainmeta
6711d977b265bbd413b83242
Zecheng He
admin_assigned
2024-10-18T12:27:38.060Z
641b7d42ec5b871c0bcf2955
/avatars/f6e24742b4befa889e72c7e5eefd6cf6.svg
He
user
zechengh
6711d977b265bbd413b83243
Zijian He
6711d977b265bbd413b83244
Albert Pumarola
6711d977b265bbd413b83245
Ali Thabet
6711d977b265bbd413b83246
Artsiom Sanakoyeu
6711d977b265bbd413b83247
Arun Mallya
admin_assigned
2024-10-18T12:28:09.467Z
6513374d2f79557a90c0f0fe
/avatars/96eadc6f0e0f91ea3d874df207a9db71.svg
Arun Mallya
user
ammp
6711d977b265bbd413b83248
Baishan Guo
admin_assigned
2024-10-18T12:28:32.745Z
66a8050feb641bdde6e60b6b
/avatars/b239d0eeaecda333a04d15fdd0a106c7.svg
Baishan Guo
user
edwardguo
6711d977b265bbd413b83249
Boris Araya
6711d977b265bbd413b8324a
Breena Kerr
6711d977b265bbd413b8324b
Carleigh Wood
6711d977b265bbd413b8324c
Ce Liu
6711d977b265bbd413b8324d
Cen Peng
6711d977b265bbd413b8324e
Dimitry Vengertsev
6711d977b265bbd413b8324f
Edgar Schonfeld
claimed_verified
2024-10-18T14:16:57.158Z
6571c662d5c6a6d3b0bdae88
/avatars/8080bcdf8b331f62383b724050189660.svg
Edgar Schoenfeld
user
edgarschoenfeld
6711d977b265bbd413b83250
Elliot Blanchard
admin_assigned
2024-10-18T12:30:11.640Z
6563cf4a718b9db5215d07b4
/avatars/20e5be0bd3fcd888959a0af0f0b462e4.svg
Elliot Blanchard
user
elliotblanchard
6711d977b265bbd413b83251
Felix Juefei-Xu
claimed_verified
2024-12-16T16:55:38.589Z
6417cf37dce1e4c0229f17b1
https://cdn-avatars.huggingface.co/v1/production/uploads/6417cf37dce1e4c0229f17b1/7h-ZCB5f4wif7TsnF-B1M.jpeg
Felix Xu
user
katanaxu
6711d977b265bbd413b83252
Fraylie Nord
6711d977b265bbd413b83253
Jeff Liang
admin_assigned
2024-10-18T12:30:33.700Z
63631be58f43a912fc7595d5
https://cdn-avatars.huggingface.co/v1/production/uploads/1667440137801-63631be58f43a912fc7595d5.jpeg
Jeff Liang
user
JeffLiang
6711d977b265bbd413b83254
John Hoffman
6711d977b265bbd413b83255
Jonas Kohler
admin_assigned
2024-10-18T12:30:01.260Z
6408a3a19e9f790c905281c2
/avatars/3517962e54bda141018e13f7e21fb1ae.svg
jonas köhler
user
Jonaskohler
6711d977b265bbd413b83256
Kaolin Fire
6711d977b265bbd413b83257
Karthik Sivakumar
6711d977b265bbd413b83258
Lawrence Chen
admin_assigned
2024-10-18T12:31:47.867Z
6303ff0c7b50dd9d0a3777e8
/avatars/1c65d5936fe0cafd1fadd1102fad28db.svg
Lawrence Chen
user
lawrencechen
6711d977b265bbd413b83259
Licheng Yu
6711d977b265bbd413b8325a
Luya Gao
6711d977b265bbd413b8325b
Markos Georgopoulos
6711d977b265bbd413b8325c
Rashel Moritz
6711d977b265bbd413b8325d
Sara K. Sampson
6711d977b265bbd413b8325e
Shikai Li
6711d977b265bbd413b8325f
Simone Parmeggiani
admin_assigned
2024-10-18T12:32:42.813Z
66a046fe7bcf5121db35e4a6
/avatars/4d407ab2245f63610c41bba602f1c0f8.svg
Simone Parmeggiani
user
ngb81
6711d977b265bbd413b83260
Steve Fine
6711d977b265bbd413b83261
Tara Fowler
6711d977b265bbd413b83262
Vladan Petrovic
admin_assigned
2024-10-18T12:24:45.907Z
6515caf71e7b9224c9d23851
/avatars/11b17ba6d4f8e7b97a1644cc551a9fe7.svg
Vladan Petrovic
user
vpetrovi
6711d977b265bbd413b83263
Yuming Du
Movie Gen: A Cast of Media Foundation Models
We present Movie Gen, a cast of foundation models that generates
high-quality, 1080p HD videos with different aspect ratios and synchronized
audio. We also show additional capabilities such as precise instruction-based
video editing and generation of personalized videos based on a user's image.
Our models set a new state-of-the-art on multiple tasks: text-to-video
synthesis, video personalization, video editing, video-to-audio generation, and
text-to-audio generation. Our largest video generation model is a 30B parameter
transformer trained with a maximum context length of 73K video tokens,
corresponding to a generated video of 16 seconds at 16 frames-per-second. We
show multiple technical innovations and simplifications on the architecture,
latent spaces, training objectives and recipes, data curation, evaluation
protocols, parallelization techniques, and inference optimizations that allow
us to reap the benefits of scaling pre-training data, model size, and training
compute for training large scale media generation models. We hope this paper
helps the research community to accelerate progress and innovation in media
generation models. All videos from this paper are available at
https://go.fb.me/MovieGenResearchVideos.
Movie Gen, a suite of foundation models, generates high-quality videos with synchronized audio, excelling in various tasks through architectural, training, and technical innovations.
transformer video tokens latent spaces training objectives data curation evaluation protocols parallelization techniques inference optimizations