CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
We introduce CogVideoX, a large-scale diffusion transformer model designed
for generating videos based on text prompts. To efficently model video data, we
propose to levearge a 3D Variational Autoencoder (VAE) to compress videos along
both spatial and temporal dimensions. To improve the text-video alignment, we
propose an expert transformer with the expert adaptive LayerNorm to facilitate
the deep fusion between the two modalities. By employing a progressive training
technique, CogVideoX is adept at producing coherent, long-duration videos
characterized by significant motions. In addition, we develop an effective
text-video data processing pipeline that includes various data preprocessing
strategies and a video captioning method. It significantly helps enhance the
performance of CogVideoX, improving both generation quality and semantic
alignment. Results show that CogVideoX demonstrates state-of-the-art
performance across both multiple machine metrics and human evaluations. The
model weights of both the 3D Causal VAE and CogVideoX are publicly available at
https://github.com/THUDM/CogVideo.
66bacf59c9b2ab14b398f761
Zhuoyi Yang
admin_assigned
2024-08-29T09:11:01.293Z
6322e02496d6c8518eb05292
/avatars/6b5d317f9ccb0046e40918c7861b4cb5.svg
zhuoyi
user
keg-yzy
66bacf59c9b2ab14b398f762
Jiayan Teng
admin_assigned
2024-08-13T08:45:51.899Z
65228733377bffdc59a10117
/avatars/6eec07553658ab22f8058caa0bfbed49.svg
tengjiayan
user
tengjiayan
66bacf59c9b2ab14b398f763
Wendi Zheng
admin_assigned
2024-08-13T08:47:07.561Z
650aaaf1634e02df56dfd231
/avatars/4643ab23721d4ed6aeb1ebbc717adc43.svg
Wendi Zheng
user
zwd125
66bacf59c9b2ab14b398f764
Ming Ding
66bacf59c9b2ab14b398f765
Shiyu Huang
admin_assigned
2024-08-13T08:47:33.688Z
6406db5cd684369027166986
https://cdn-avatars.huggingface.co/v1/production/uploads/6406db5cd684369027166986/Zl-orrGcbY0RbfjfKszn1.jpeg
Shiyu Huang
user
ShiyuHuang
66bacf59c9b2ab14b398f766
Jiazheng Xu
admin_assigned
2024-08-13T08:47:56.691Z
62d7b131f6e8ba66107af761
/avatars/f1c5df47aef69c824fd166722df8f670.svg
Jiazheng Xu
user
xujz0703
66bacf59c9b2ab14b398f767
Yuanming Yang
admin_assigned
2024-08-13T08:48:10.864Z
659b6c50b0f43ed69fe09d56
/avatars/8ea56c56263595a9f7555f2c2520641a.svg
杨远明
user
yangyuanming
66bacf59c9b2ab14b398f768
Wenyi Hong
admin_assigned
2024-08-13T08:48:17.739Z
62ecd24cb8764c7738ef2793
/avatars/c1b80b5c55f9d652c1aaac7919e1fa32.svg
Wenyi Hong
user
wenyi
66bacf59c9b2ab14b398f769
Xiaohan Zhang
66bacf59c9b2ab14b398f76a
Guanyu Feng
admin_assigned
2024-08-13T08:48:57.709Z
64d996d6bcab729cb400cb70
/avatars/3db57c8b643fba8302ed39d8cf0f4ddb.svg
Guanyu Feng
user
jiguanglizipao
66bacf59c9b2ab14b398f76b
Da Yin
admin_assigned
2024-08-13T08:49:04.537Z
634e4670a51d5df8c2d92fce
/avatars/c52d7150b4de6a2eb2d83b345d35cbc2.svg
Da Yin
user
DaYin
66bacf59c9b2ab14b398f76c
Xiaotao Gu
claimed_verified
2025-07-02T09:25:11.545Z
6438bca4a5e10f6d58694b47
/avatars/3aeb25fbc73c5cab1265e13d11adfb76.svg
Xiaotao Gu
user
xgeric
66bacf59c9b2ab14b398f76d
Yuxuan Zhang
claimed_verified
2024-08-28T15:29:07.523Z
643507d1ce04fdb57e9d7e05
https://cdn-avatars.huggingface.co/v1/production/uploads/643507d1ce04fdb57e9d7e05/QuCFAO3v7G3LrVGusqFj1.png
zR
user
ZHANGYUXUAN-zR
66bacf59c9b2ab14b398f76e
Weihan Wang
claimed_verified
2025-07-02T12:23:31.640Z
63403c75b8b51e0098d3555c
/avatars/85725280b63788e387fe73319a54164d.svg
王维汉
user
mactavish91
66bacf59c9b2ab14b398f76f
Yean Cheng
admin_assigned
2024-08-13T08:50:14.803Z
65acc5afe2a2c8635614de43
/avatars/c5fce792792cc0b52ed7475d72460c58.svg
Yean Cheng
user
LiquidAmmonia
66bacf59c9b2ab14b398f770
Ting Liu
66bacf59c9b2ab14b398f771
Bin Xu
66bacf59c9b2ab14b398f772
Yuxiao Dong
admin_assigned
2024-08-13T08:51:14.639Z
640e73bdfdeaae1390857b62
/avatars/cd6779e30f716002a7838ed93d5c0754.svg
Yuxiao Dong
user
yuxiaod
66bacf59c9b2ab14b398f773
Jie Tang
admin_assigned
2024-08-13T08:51:32.360Z
640dff05474aa6f89556677e
/avatars/1b4591c7322d649c797b3125148f1915.svg
Jie Tang
user
jerytang
CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
We introduce CogVideoX, a large-scale diffusion transformer model designed
for generating videos based on text prompts. To efficently model video data, we
propose to levearge a 3D Variational Autoencoder (VAE) to compress videos along
both spatial and temporal dimensions. To improve the text-video alignment, we
propose an expert transformer with the expert adaptive LayerNorm to facilitate
the deep fusion between the two modalities. By employing a progressive training
technique, CogVideoX is adept at producing coherent, long-duration videos
characterized by significant motions. In addition, we develop an effective
text-video data processing pipeline that includes various data preprocessing
strategies and a video captioning method. It significantly helps enhance the
performance of CogVideoX, improving both generation quality and semantic
alignment. Results show that CogVideoX demonstrates state-of-the-art
performance across both multiple machine metrics and human evaluations. The
model weights of both the 3D Causal VAE and CogVideoX are publicly available at
https://github.com/THUDM/CogVideo.
CogVideoX is a large-scale diffusion transformer model using a 3D Variational Autoencoder and expert transformer for generating high-quality, coherent videos from text prompts.
diffusion transformer 3D Variational Autoencoder expert transformer expert adaptive LayerNorm progressive training video captioning