The Best of Both Worlds: Integrating Language Models and Diffusion
  Models for Video Generation
Recent advancements in text-to-video (T2V) generation have been driven by two
competing paradigms: autoregressive language models and diffusion models.
However, each paradigm has intrinsic limitations: language models struggle with
visual quality and error accumulation, while diffusion models lack semantic
understanding and causal modeling. In this work, we propose LanDiff, a hybrid
framework that synergizes the strengths of both paradigms through
coarse-to-fine generation. Our architecture introduces three key innovations:
(1) a semantic tokenizer that compresses 3D visual features into compact 1D
discrete representations through efficient semantic compression, achieving a
sim14,000times compression ratio; (2) a language model that generates
semantic tokens with high-level semantic relationships; (3) a streaming
diffusion model that refines coarse semantics into high-fidelity videos.
Experiments show that LanDiff, a 5B model, achieves a score of 85.43 on the
VBench T2V benchmark, surpassing the state-of-the-art open-source models
Hunyuan Video (13B) and other commercial models such as Sora, Keling, and
Hailuo. Furthermore, our model also achieves state-of-the-art performance in
long video generation, surpassing other open-source models in this field. Our
demo can be viewed at https://landiff.github.io/.
67ca7b8a2a2c299d98944909
Aoxiong Yin
67ca7b8a2a2c299d9894490a
Kai Shen
67ca7b8a2a2c299d9894490b
Yichong Leng
admin_assigned
2025-03-07T09:41:57.733Z
64a0347b528a9bbe59d6e08c
/avatars/6dd0bad84d711d1048a0a4169e621773.svg
Yichong Leng
user
ustcscallion
67ca7b8a2a2c299d9894490c
Xu Tan
claimed_verified
2025-03-12T08:43:46.715Z
5f1040b6e9d71719e3be71d2
/avatars/a2f28940236ae625ed3810ad62e343ff.svg
Xu Tan
user
xutan
67ca7b8a2a2c299d9894490d
Xinyu Zhou
67ca7b8a2a2c299d9894490e
Juncheng Li
admin_assigned
2025-03-07T09:42:06.818Z
67bc247b593452cc18965cb1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/EA3kTYaaff0Hr7-dGiOOj.png
JUNCHENG LI
user
JunchengLi
67ca7b8a2a2c299d9894490f
Siliang Tang
The Best of Both Worlds: Integrating Language Models and Diffusion
  Models for Video Generation
Recent advancements in text-to-video (T2V) generation have been driven by two
competing paradigms: autoregressive language models and diffusion models.
However, each paradigm has intrinsic limitations: language models struggle with
visual quality and error accumulation, while diffusion models lack semantic
understanding and causal modeling. In this work, we propose LanDiff, a hybrid
framework that synergizes the strengths of both paradigms through
coarse-to-fine generation. Our architecture introduces three key innovations:
(1) a semantic tokenizer that compresses 3D visual features into compact 1D
discrete representations through efficient semantic compression, achieving a
sim14,000times compression ratio; (2) a language model that generates
semantic tokens with high-level semantic relationships; (3) a streaming
diffusion model that refines coarse semantics into high-fidelity videos.
Experiments show that LanDiff, a 5B model, achieves a score of 85.43 on the
VBench T2V benchmark, surpassing the state-of-the-art open-source models
Hunyuan Video (13B) and other commercial models such as Sora, Keling, and
Hailuo. Furthermore, our model also achieves state-of-the-art performance in
long video generation, surpassing other open-source models in this field. Our
demo can be viewed at https://landiff.github.io/.
LanDiff, a hybrid of autoregressive language models and diffusion models, achieves high visual quality and semantic understanding in text-to-video generation, significantly outperforming existing models.
autoregressive language models diffusion models semantic tokenizer efficient semantic compression high-level semantic relationships streaming diffusion model VBench T2V benchmark