xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed
  Representations
We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of
producing realistic scenes from textual descriptions. Building on recent
advancements, such as OpenAI's Sora, we explore the latent diffusion model
(LDM) architecture and introduce a video variational autoencoder (VidVAE).
VidVAE compresses video data both spatially and temporally, significantly
reducing the length of visual tokens and the computational demands associated
with generating long-sequence videos. To further address the computational
costs, we propose a divide-and-merge strategy that maintains temporal
consistency across video segments. Our Diffusion Transformer (DiT) model
incorporates spatial and temporal self-attention layers, enabling robust
generalization across different timeframes and aspect ratios. We have devised a
data processing pipeline from the very beginning and collected over 13M
high-quality video-text pairs. The pipeline includes multiple steps such as
clipping, text detection, motion estimation, aesthetics scoring, and dense
captioning based on our in-house video-LLM model. Training the VidVAE and DiT
models required approximately 40 and 642 H100 days, respectively. Our model
supports over 14-second 720p video generation in an end-to-end way and
demonstrates competitive performance against state-of-the-art T2V models.
66c7ef2596661a3c4eaa9fb5
Can Qin
claimed_verified
2025-10-10T10:15:24.865Z
66a9bdfc35c89692442ba4b7
/avatars/ad87d1d2d81775eeb0920bf3ebe08cc2.svg
Can Qin
user
canqin001
66c7ef2596661a3c4eaa9fb6
Congying Xia
66c7ef2596661a3c4eaa9fb7
Krithika Ramakrishnan
66c7ef2596661a3c4eaa9fb8
Michael Ryoo
66c7ef2596661a3c4eaa9fb9
Lifu Tu
66c7ef2596661a3c4eaa9fba
Yihao Feng
66c7ef2596661a3c4eaa9fbb
Manli Shu
66c7ef2596661a3c4eaa9fbc
Honglu Zhou
66c7ef2596661a3c4eaa9fbd
Anas Awadalla
claimed_verified
2024-08-29T07:23:43.580Z
60f1ab3afe0d78e01037eeb1
/avatars/f784fa423fd84fffb4683fa837ffc5a3.svg
Anas Awadalla
user
anas-awadalla
66c7ef2596661a3c4eaa9fbe
Jun Wang
66c7ef2596661a3c4eaa9fbf
Senthil Purushwalkam
66c7ef2596661a3c4eaa9fc0
Le Xue
66c7ef2596661a3c4eaa9fc1
Yingbo Zhou
66c7ef2596661a3c4eaa9fc2
Huan Wang
66c7ef2596661a3c4eaa9fc3
Silvio Savarese
66c7ef2596661a3c4eaa9fc4
Juan Carlos Niebles
66c7ef2596661a3c4eaa9fc5
Zeyuan Chen
66c7ef2596661a3c4eaa9fc6
Ran Xu
claimed_verified
2025-05-15T10:41:43.287Z
6465c4c863e7e09dd02e3e1b
/avatars/200b029184d2616f98296a2c212f0785.svg
Ran Xu
user
xurantju
66c7ef2596661a3c4eaa9fc7
Caiming Xiong
xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed
  Representations
We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of
producing realistic scenes from textual descriptions. Building on recent
advancements, such as OpenAI's Sora, we explore the latent diffusion model
(LDM) architecture and introduce a video variational autoencoder (VidVAE).
VidVAE compresses video data both spatially and temporally, significantly
reducing the length of visual tokens and the computational demands associated
with generating long-sequence videos. To further address the computational
costs, we propose a divide-and-merge strategy that maintains temporal
consistency across video segments. Our Diffusion Transformer (DiT) model
incorporates spatial and temporal self-attention layers, enabling robust
generalization across different timeframes and aspect ratios. We have devised a
data processing pipeline from the very beginning and collected over 13M
high-quality video-text pairs. The pipeline includes multiple steps such as
clipping, text detection, motion estimation, aesthetics scoring, and dense
captioning based on our in-house video-LLM model. Training the VidVAE and DiT
models required approximately 40 and 642 H100 days, respectively. Our model
supports over 14-second 720p video generation in an end-to-end way and
demonstrates competitive performance against state-of-the-art T2V models.
The xGen-VideoSyn-1 model generates realistic videos from text using a Latent Diffusion Model and Video Variational Autoencoder, with a spatial and temporal divide-and-merge strategy and Diffusion Transformer.
latent diffusion model VVVAE spatial compression temporal compression visual tokens computational demands divide-and-merge strategy temporal consistency Diffusion Transformer spatial self-attention temporal self-attention robust generalization text-to-video video-text pairs clippng text detection motion estimation aesthetics scoring dense captioning video-LLM model