UniFusion: Vision-Language Model as Unified Encoder in Image Generation
Although recent advances in visual generation have been remarkable, most
existing architectures still depend on distinct encoders for images and text.
This separation constrains diffusion models' ability to perform cross-modal
reasoning and knowledge transfer. Prior attempts to bridge this gap often use
the last layer information from VLM, employ multiple visual encoders, or train
large unified models jointly for text and image generation, which demands
substantial computational resources and large-scale data, limiting its
accessibility.We present UniFusion, a diffusion-based generative model
conditioned on a frozen large vision-language model (VLM) that serves as a
unified multimodal encoder. At the core of UniFusion is the Layerwise Attention
Pooling (LAP) mechanism that extracts both high level semantics and low level
details from text and visual tokens of a frozen VLM to condition a diffusion
generative model. We demonstrate that LAP outperforms other shallow fusion
architectures on text-image alignment for generation and faithful transfer of
visual information from VLM to the diffusion model which is key for editing. We
propose VLM-Enabled Rewriting Injection with Flexibile Inference (VERIFI),
which conditions a diffusion transformer (DiT) only on the text tokens
generated by the VLM during in-model prompt rewriting. VERIFI combines the
alignment of the conditioning distribution with the VLM's reasoning
capabilities for increased capabilities and flexibility at inference. In
addition, finetuning on editing task not only improves text-image alignment for
generation, indicative of cross-modality knowledge transfer, but also exhibits
tremendous generalization capabilities. Our model when trained on single image
editing, zero-shot generalizes to multiple image references further motivating
the unified encoder design of UniFusion.
68eefdae486b78128f0e336d
Kevin Li
claimed_verified
2025-10-20T12:13:51.935Z
645723a3bfdf9c63ce33959a
https://cdn-avatars.huggingface.co/v1/production/uploads/645723a3bfdf9c63ce33959a/9k6ay8qKCw5s0nzuLWo1j.png
Kevin Li
user
Curiouskid
68eefdae486b78128f0e336e
Manuel Brack
claimed_verified
2025-10-15T07:07:40.869Z
62fa1d95e8c9c532aa75331c
https://cdn-avatars.huggingface.co/v1/production/uploads/62fa1d95e8c9c532aa75331c/WFfk_n8gOj845pSkfdazA.jpeg
Manuel Brack
user
mbrack
68eefdae486b78128f0e336f
Sudeep Katakol
68eefdae486b78128f0e3370
Hareesh Ravi
claimed_verified
2025-10-16T06:41:55.379Z
630d41f3e57da2042090669d
/avatars/bc569dfd2d414dac1a5609b01f795da1.svg
Hareesh Ravi
user
haravi
68eefdae486b78128f0e3371
Ajinkya Kale
UniFusion: Vision-Language Model as Unified Encoder in Image Generation
Although recent advances in visual generation have been remarkable, most
existing architectures still depend on distinct encoders for images and text.
This separation constrains diffusion models' ability to perform cross-modal
reasoning and knowledge transfer. Prior attempts to bridge this gap often use
the last layer information from VLM, employ multiple visual encoders, or train
large unified models jointly for text and image generation, which demands
substantial computational resources and large-scale data, limiting its
accessibility.We present UniFusion, a diffusion-based generative model
conditioned on a frozen large vision-language model (VLM) that serves as a
unified multimodal encoder. At the core of UniFusion is the Layerwise Attention
Pooling (LAP) mechanism that extracts both high level semantics and low level
details from text and visual tokens of a frozen VLM to condition a diffusion
generative model. We demonstrate that LAP outperforms other shallow fusion
architectures on text-image alignment for generation and faithful transfer of
visual information from VLM to the diffusion model which is key for editing. We
propose VLM-Enabled Rewriting Injection with Flexibile Inference (VERIFI),
which conditions a diffusion transformer (DiT) only on the text tokens
generated by the VLM during in-model prompt rewriting. VERIFI combines the
alignment of the conditioning distribution with the VLM's reasoning
capabilities for increased capabilities and flexibility at inference. In
addition, finetuning on editing task not only improves text-image alignment for
generation, indicative of cross-modality knowledge transfer, but also exhibits
tremendous generalization capabilities. Our model when trained on single image
editing, zero-shot generalizes to multiple image references further motivating
the unified encoder design of UniFusion.