Transfusion: Predict the Next Token and Diffuse Images with One
  Multi-Modal Model
We introduce Transfusion, a recipe for training a multi-modal model over
discrete and continuous data. Transfusion combines the language modeling loss
function (next token prediction) with diffusion to train a single transformer
over mixed-modality sequences. We pretrain multiple Transfusion models up to 7B
parameters from scratch on a mixture of text and image data, establishing
scaling laws with respect to a variety of uni- and cross-modal benchmarks. Our
experiments show that Transfusion scales significantly better than quantizing
images and training a language model over discrete image tokens. By introducing
modality-specific encoding and decoding layers, we can further improve the
performance of Transfusion models, and even compress each image to just 16
patches. We further demonstrate that scaling our Transfusion recipe to 7B
parameters and 2T multi-modal tokens produces a model that can generate images
and text on a par with similar scale diffusion models and language models,
reaping the benefits of both worlds.
66c556167690afd2f4c47e12
Chunting Zhou
66c556167690afd2f4c47e13
Lili Yu
66c556167690afd2f4c47e14
Arun Babu
66c556167690afd2f4c47e15
Kushal Tirumala
66c556167690afd2f4c47e16
Michihiro Yasunaga
claimed_verified
2025-02-21T16:39:16.320Z
621e9388345a1d9ab65391c3
/avatars/d009c811dc63224f8eba1889fefad04b.svg
Michihiro Yasunaga
user
michiyasunaga
66c556167690afd2f4c47e17
Leonid Shamis
66c556167690afd2f4c47e18
Jacob Kahn
66c556167690afd2f4c47e19
Xuezhe Ma
66c556167690afd2f4c47e1a
Luke Zettlemoyer
66c556167690afd2f4c47e1b
Omer Levy
Transfusion: Predict the Next Token and Diffuse Images with One
  Multi-Modal Model
We introduce Transfusion, a recipe for training a multi-modal model over
discrete and continuous data. Transfusion combines the language modeling loss
function (next token prediction) with diffusion to train a single transformer
over mixed-modality sequences. We pretrain multiple Transfusion models up to 7B
parameters from scratch on a mixture of text and image data, establishing
scaling laws with respect to a variety of uni- and cross-modal benchmarks. Our
experiments show that Transfusion scales significantly better than quantizing
images and training a language model over discrete image tokens. By introducing
modality-specific encoding and decoding layers, we can further improve the
performance of Transfusion models, and even compress each image to just 16
patches. We further demonstrate that scaling our Transfusion recipe to 7B
parameters and 2T multi-modal tokens produces a model that can generate images
and text on a par with similar scale diffusion models and language models,
reaping the benefits of both worlds.
Transfusion combines language modeling and diffusion training to develop a multi-modal transformer model that scales effectively across text and image data, matching the performance of specialized models.
language modeling diffusion transformer multi-modal uni-modal cross-modal benchmarks quantizing discrete image tokens modality-specific encoding modality-specific decoding image generation