Multimodal Representation Alignment for Image Generation: Text-Image
  Interleaved Control Is Easier Than You Think
The field of advanced text-to-image generation is witnessing the emergence of
unified frameworks that integrate powerful text encoders, such as CLIP and T5,
with Diffusion Transformer backbones. Although there have been efforts to
control output images with additional conditions, like canny and depth map, a
comprehensive framework for arbitrary text-image interleaved control is still
lacking. This gap is especially evident when attempting to merge concepts or
visual elements from multiple images in the generation process. To mitigate the
gap, we conducted preliminary experiments showing that large multimodal models
(LMMs) offer an effective shared representation space, where image and text can
be well-aligned to serve as a condition for external diffusion models. Based on
this discovery, we propose Dream Engine, an efficient and unified framework
designed for arbitrary text-image interleaved control in image generation
models. Building on powerful text-to-image models like SD3.5, we replace the
original text-only encoders by incorporating versatile multimodal information
encoders such as QwenVL. Our approach utilizes a two-stage training paradigm,
consisting of joint text-image alignment and multimodal interleaved instruction
tuning. Our experiments demonstrate that this training method is effective,
achieving a 0.69 overall score on the GenEval benchmark, and matching the
performance of state-of-the-art text-to-image models like SD3.5 and FLUX.
67c17b8f60206395233b7e46
Liang Chen
admin_assigned
2025-02-28T12:34:40.397Z
658c481dd1c8b106727a8b73
/avatars/d34a7a62c3a524e5fdd2d5994348db58.svg
Liang Chen
user
liangchen-ms
67c17b8f60206395233b7e47
Shuai Bai
admin_assigned
2025-02-28T12:33:50.796Z
63451cf0a05b51f7ded25505
/avatars/dec4bbee4a82b773fc58dfc2dce9dbeb.svg
shuai bai
user
bluelike
67c17b8f60206395233b7e48
Wenhao Chai
admin_assigned
2025-02-28T12:33:57.358Z
637c7503fe115289cfecbe6b
https://cdn-avatars.huggingface.co/v1/production/uploads/1676361945047-637c7503fe115289cfecbe6b.jpeg
Wenhao Chai
user
wchai
67c17b8f60206395233b7e49
Weichu Xie
admin_assigned
2025-02-28T12:34:48.933Z
678609789a285d232ee14157
/avatars/a6cb2c571d9ef6deb0b1659f754afe7f.svg
Weichu Xie
user
akarinmoe
67c17b8f60206395233b7e4a
Haozhe Zhao
claimed_verified
2025-05-26T08:42:16.039Z
63ff09a02098b9ad105a09f6
/avatars/4409ca5d320050cf4c3df05962c7ff58.svg
Hans Zhao
user
BleachNick
67c17b8f60206395233b7e4b
Leon Vinci
67c17b8f60206395233b7e4c
Junyang Lin
admin_assigned
2025-02-28T12:34:24.275Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
67c17b8f60206395233b7e4d
Baobao Chang
Multimodal Representation Alignment for Image Generation: Text-Image
  Interleaved Control Is Easier Than You Think
The field of advanced text-to-image generation is witnessing the emergence of
unified frameworks that integrate powerful text encoders, such as CLIP and T5,
with Diffusion Transformer backbones. Although there have been efforts to
control output images with additional conditions, like canny and depth map, a
comprehensive framework for arbitrary text-image interleaved control is still
lacking. This gap is especially evident when attempting to merge concepts or
visual elements from multiple images in the generation process. To mitigate the
gap, we conducted preliminary experiments showing that large multimodal models
(LMMs) offer an effective shared representation space, where image and text can
be well-aligned to serve as a condition for external diffusion models. Based on
this discovery, we propose Dream Engine, an efficient and unified framework
designed for arbitrary text-image interleaved control in image generation
models. Building on powerful text-to-image models like SD3.5, we replace the
original text-only encoders by incorporating versatile multimodal information
encoders such as QwenVL. Our approach utilizes a two-stage training paradigm,
consisting of joint text-image alignment and multimodal interleaved instruction
tuning. Our experiments demonstrate that this training method is effective,
achieving a 0.69 overall score on the GenEval benchmark, and matching the
performance of state-of-the-art text-to-image models like SD3.5 and FLUX.
A unified framework called Dream Engine integrates powerful text-to-image models with versatile multimodal encoders to enable arbitrary text-image interleaved control in image generation.
CLIP T5 Diffusion Transformer canny depth map large multimodal models LMMs Dream Engine text-only encoders QwenVL joint text-image alignment multimodal interleaved instruction tuning GenEval benchmark SD3.5 FLUX