Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with
  Patch-level CLIP Latents
There is growing interest in integrating high-fidelity visual synthesis
capabilities into large language models (LLMs) without compromising their
strong reasoning capabilities. Existing methods that directly train LLMs or
bridge LLMs and diffusion models usually suffer from costly training since the
backbone LLMs have not seen image representations during pretraining. We
present Bifrost-1, a unified framework that bridges pretrained multimodal LLMs
(MLLMs) and diffusion models using patch-level CLIP image embeddings as latent
variables, which are natively aligned with the MLLM's CLIP visual encoder.
These patch-level image embeddings are integrated into the diffusion model with
a lightweight adaptation of its ControlNet. To retain the original multimodal
reasoning capabilities of MLLMs, we equip the MLLM with a visual generation
branch initialized from the original MLLM parameters when predicting the
patch-level image embeddings. By seamlessly integrating pretrained MLLMs and
diffusion models with patch-level CLIP latents, our framework enables
high-fidelity controllable image generation with significant training
efficiency. Our experiments demonstrate that Bifrost-1 achieves comparable or
better performance than previous methods in terms of visual fidelity and
multimodal understanding, with substantially lower compute during training. We
also provide comprehensive ablation studies showing the effectiveness of our
design choices.
689a1735fab6fdd2e52ac3a0
Han Lin
689a1735fab6fdd2e52ac3a1
Jaemin Cho
claimed_verified
2025-08-13T07:21:47.747Z
5ffe32d8942cf3533d364449
https://cdn-avatars.huggingface.co/v1/production/uploads/1654821969191-5ffe32d8942cf3533d364449.jpeg
Jaemin Cho
user
j-min
689a1735fab6fdd2e52ac3a2
Amir Zadeh
689a1735fab6fdd2e52ac3a3
Chuan Li
689a1735fab6fdd2e52ac3a4
Mohit Bansal
Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with
  Patch-level CLIP Latents
There is growing interest in integrating high-fidelity visual synthesis
capabilities into large language models (LLMs) without compromising their
strong reasoning capabilities. Existing methods that directly train LLMs or
bridge LLMs and diffusion models usually suffer from costly training since the
backbone LLMs have not seen image representations during pretraining. We
present Bifrost-1, a unified framework that bridges pretrained multimodal LLMs
(MLLMs) and diffusion models using patch-level CLIP image embeddings as latent
variables, which are natively aligned with the MLLM's CLIP visual encoder.
These patch-level image embeddings are integrated into the diffusion model with
a lightweight adaptation of its ControlNet. To retain the original multimodal
reasoning capabilities of MLLMs, we equip the MLLM with a visual generation
branch initialized from the original MLLM parameters when predicting the
patch-level image embeddings. By seamlessly integrating pretrained MLLMs and
diffusion models with patch-level CLIP latents, our framework enables
high-fidelity controllable image generation with significant training
efficiency. Our experiments demonstrate that Bifrost-1 achieves comparable or
better performance than previous methods in terms of visual fidelity and
multimodal understanding, with substantially lower compute during training. We
also provide comprehensive ablation studies showing the effectiveness of our
design choices.
Bifrost-1 integrates pretrained multimodal LLMs and diffusion models using patch-level CLIP embeddings to enable efficient high-fidelity image generation with strong multimodal reasoning.
LLMs diffusion models patch-level CLIP image embeddings latent variables CLIP visual encoder ControlNet visual generation branch multimodal reasoning visual fidelity multimodal understanding