Causal Diffusion Transformers for Generative Modeling
We introduce Causal Diffusion as the autoregressive (AR) counterpart of
Diffusion models. It is a next-token(s) forecasting framework that is friendly
to both discrete and continuous modalities and compatible with existing
next-token prediction models like LLaMA and GPT. While recent works attempt to
combine diffusion with AR models, we show that introducing sequential
factorization to a diffusion model can substantially improve its performance
and enables a smooth transition between AR and diffusion generation modes.
Hence, we propose CausalFusion - a decoder-only transformer that
dual-factorizes data across sequential tokens and diffusion noise levels,
leading to state-of-the-art results on the ImageNet generation benchmark while
also enjoying the AR advantage of generating an arbitrary number of tokens for
in-context reasoning. We further demonstrate CausalFusion's multimodal
capabilities through a joint image generation and captioning model, and
showcase CausalFusion's ability for zero-shot in-context image manipulations.
We hope that this work could provide the community with a fresh perspective on
training multimodal models over discrete and continuous data.
67610887df97bdafe87c37aa
Chaorui Deng
67610887df97bdafe87c37ab
Deyao Zh
67610887df97bdafe87c37ac
Kunchang Li
claimed_verified
2024-12-17T08:03:39.007Z
61fb81006374891646732f37
https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
Kunchang Li
user
Andy1621
67610887df97bdafe87c37ad
Shi Guan
67610887df97bdafe87c37ae
Haoqi Fan
Causal Diffusion Transformers for Generative Modeling
We introduce Causal Diffusion as the autoregressive (AR) counterpart of
Diffusion models. It is a next-token(s) forecasting framework that is friendly
to both discrete and continuous modalities and compatible with existing
next-token prediction models like LLaMA and GPT. While recent works attempt to
combine diffusion with AR models, we show that introducing sequential
factorization to a diffusion model can substantially improve its performance
and enables a smooth transition between AR and diffusion generation modes.
Hence, we propose CausalFusion - a decoder-only transformer that
dual-factorizes data across sequential tokens and diffusion noise levels,
leading to state-of-the-art results on the ImageNet generation benchmark while
also enjoying the AR advantage of generating an arbitrary number of tokens for
in-context reasoning. We further demonstrate CausalFusion's multimodal
capabilities through a joint image generation and captioning model, and
showcase CausalFusion's ability for zero-shot in-context image manipulations.
We hope that this work could provide the community with a fresh perspective on
training multimodal models over discrete and continuous data.
Causal Diffusion combines autoregressive models with diffusion techniques to improve generation performance and multimodal capabilities, achieving state-of-the-art results in image generation and zero-shot image manipulations.
Causal Diffusion autoregressive (AR) Diffusion models next-token(s) forecasting discrete modalities continuous modalities next-token prediction models sequential factorization CausalFusion decoder-only transformer multimodal capabilities image generation benchmark in-context reasoning zero-shot in-context image manipulations