Multimodal Latent Language Modeling with Next-Token Diffusion
Multimodal generative models require a unified approach to handle both
discrete data (e.g., text and code) and continuous data (e.g., image, audio,
video). In this work, we propose Latent Language Modeling (LatentLM), which
seamlessly integrates continuous and discrete data using causal Transformers.
Specifically, we employ a variational autoencoder (VAE) to represent continuous
data as latent vectors and introduce next-token diffusion for autoregressive
generation of these vectors. Additionally, we develop sigma-VAE to address
the challenges of variance collapse, which is crucial for autoregressive
modeling. Extensive experiments demonstrate the effectiveness of LatentLM
across various modalities. In image generation, LatentLM surpasses Diffusion
Transformers in both performance and scalability. When integrated into
multimodal large language models, LatentLM provides a general-purpose interface
that unifies multimodal generation and understanding. Experimental results show
that LatentLM achieves favorable performance compared to Transfusion and vector
quantized models in the setting of scaling up training tokens. In
text-to-speech synthesis, LatentLM outperforms the state-of-the-art VALL-E 2
model in speaker similarity and robustness, while requiring 10x fewer decoding
steps. The results establish LatentLM as a highly effective and scalable
approach to advance large multimodal models.
675b17aa378af168bbfd66fa
Yutao Sun
admin_assigned
2024-12-16T10:41:50.828Z
6300ef4779c5ddbc6cf83e1a
https://cdn-avatars.huggingface.co/v1/production/uploads/1661005591657-noauth.jpeg
Yutao Sun
user
sunyt32
675b17aa378af168bbfd66fb
Hangbo Bao
admin_assigned
2024-12-16T10:41:44.329Z
62c6981c9db11473f08b0c87
https://cdn-avatars.huggingface.co/v1/production/uploads/62c6981c9db11473f08b0c87/PKnnQ1SOrMuB4vf2cDDdG.jpeg
Hangbo Bao
user
addf400
675b17aa378af168bbfd66fc
Wenhui Wang
675b17aa378af168bbfd66fd
Zhiliang Peng
claimed_verified
2025-09-01T07:56:58.462Z
646c408336505117e22f4b36
/avatars/8201da3c4ec51c8e113446f9578d44f6.svg
zhiliang
user
zzliang
675b17aa378af168bbfd66fe
Li Dong
claimed_verified
2024-12-13T15:23:05.164Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
user
unilm
675b17aa378af168bbfd66ff
Shaohan Huang
claimed_verified
2025-08-28T08:58:54.095Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
675b17aa378af168bbfd6700
Jianyong Wang
675b17aa378af168bbfd6701
Furu Wei
admin_assigned
2024-12-16T10:39:56.365Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Multimodal Latent Language Modeling with Next-Token Diffusion
Multimodal generative models require a unified approach to handle both
discrete data (e.g., text and code) and continuous data (e.g., image, audio,
video). In this work, we propose Latent Language Modeling (LatentLM), which
seamlessly integrates continuous and discrete data using causal Transformers.
Specifically, we employ a variational autoencoder (VAE) to represent continuous
data as latent vectors and introduce next-token diffusion for autoregressive
generation of these vectors. Additionally, we develop sigma-VAE to address
the challenges of variance collapse, which is crucial for autoregressive
modeling. Extensive experiments demonstrate the effectiveness of LatentLM
across various modalities. In image generation, LatentLM surpasses Diffusion
Transformers in both performance and scalability. When integrated into
multimodal large language models, LatentLM provides a general-purpose interface
that unifies multimodal generation and understanding. Experimental results show
that LatentLM achieves favorable performance compared to Transfusion and vector
quantized models in the setting of scaling up training tokens. In
text-to-speech synthesis, LatentLM outperforms the state-of-the-art VALL-E 2
model in speaker similarity and robustness, while requiring 10x fewer decoding
steps. The results establish LatentLM as a highly effective and scalable
approach to advance large multimodal models.
LatentLM integrates continuous and discrete data using causal Transformers, VAEs, and next-token diffusion, achieving superior performance in multimodal tasks like image generation, large language model integration, and text-to-speech synthesis.
LatentLM causal Transformers variational autoencoder (VAE) next-token diffusion $\sigma$-VAE variance collapse Diffusion Transformers Transfusion vector quantized models VALL-E 2 speaker similarity robustness decoding steps