EMMA: Your Text-to-Image Diffusion Model Can Secretly Accept Multi-Modal
  Prompts
Recent advancements in image generation have enabled the creation of
high-quality images from text conditions. However, when facing multi-modal
conditions, such as text combined with reference appearances, existing methods
struggle to balance multiple conditions effectively, typically showing a
preference for one modality over others. To address this challenge, we
introduce EMMA, a novel image generation model accepting multi-modal prompts
built upon the state-of-the-art text-to-image (T2I) diffusion model, ELLA. EMMA
seamlessly incorporates additional modalities alongside text to guide image
generation through an innovative Multi-modal Feature Connector design, which
effectively integrates textual and supplementary modal information using a
special attention mechanism. By freezing all parameters in the original T2I
diffusion model and only adjusting some additional layers, we reveal an
interesting finding that the pre-trained T2I diffusion model can secretly
accept multi-modal prompts. This interesting property facilitates easy
adaptation to different existing frameworks, making EMMA a flexible and
effective tool for producing personalized and context-aware images and even
videos. Additionally, we introduce a strategy to assemble learned EMMA modules
to produce images conditioned on multiple modalities simultaneously,
eliminating the need for additional training with mixed multi-modal prompts.
Extensive experiments demonstrate the effectiveness of EMMA in maintaining high
fidelity and detail in generated images, showcasing its potential as a robust
solution for advanced multi-modal conditional image generation tasks.
666ba936435a489ffd15a8f2
Yucheng Han
claimed_verified
2024-06-14T07:22:43.785Z
64c38f6a7641afa6ef7d40a9
/avatars/8277e7c13f80973bdb9a2dae36e6e728.svg
Han Yucheng
user
listen2you
666ba936435a489ffd15a8f3
Rui Wang
claimed_verified
2024-06-14T07:22:46.011Z
630443483926de1f7ec6bda0
/avatars/e8e95ab0d03f58c61846ccc1318502fe.svg
Rui Wang
user
budui
666ba936435a489ffd15a8f4
Chi Zhang
claimed_verified
2024-08-08T07:42:51.865Z
64196320ed725fef64419c2a
/avatars/96feb22fb5e8931d6c9e0ea06148266f.svg
Chi Zhang
user
DrChiZhang
666ba936435a489ffd15a8f5
Juntao Hu
admin_assigned
2024-06-14T11:49:22.802Z
664edcade24897b66ca6e5b0
/avatars/a122c2e56e4927f9e45f125d51d3d836.svg
Juntao Hu
user
TBI805
666ba936435a489ffd15a8f6
Pei Cheng
666ba936435a489ffd15a8f7
Bin Fu
666ba936435a489ffd15a8f8
Hanwang Zhang
EMMA: Your Text-to-Image Diffusion Model Can Secretly Accept Multi-Modal
  Prompts
Recent advancements in image generation have enabled the creation of
high-quality images from text conditions. However, when facing multi-modal
conditions, such as text combined with reference appearances, existing methods
struggle to balance multiple conditions effectively, typically showing a
preference for one modality over others. To address this challenge, we
introduce EMMA, a novel image generation model accepting multi-modal prompts
built upon the state-of-the-art text-to-image (T2I) diffusion model, ELLA. EMMA
seamlessly incorporates additional modalities alongside text to guide image
generation through an innovative Multi-modal Feature Connector design, which
effectively integrates textual and supplementary modal information using a
special attention mechanism. By freezing all parameters in the original T2I
diffusion model and only adjusting some additional layers, we reveal an
interesting finding that the pre-trained T2I diffusion model can secretly
accept multi-modal prompts. This interesting property facilitates easy
adaptation to different existing frameworks, making EMMA a flexible and
effective tool for producing personalized and context-aware images and even
videos. Additionally, we introduce a strategy to assemble learned EMMA modules
to produce images conditioned on multiple modalities simultaneously,
eliminating the need for additional training with mixed multi-modal prompts.
Extensive experiments demonstrate the effectiveness of EMMA in maintaining high
fidelity and detail in generated images, showcasing its potential as a robust
solution for advanced multi-modal conditional image generation tasks.
EMMA enhances multi-modal image generation by integrating text and visual inputs into a diffusion model through a Multi-modal Feature Connector and attention mechanism, allowing for high-quality, context-aware image and video creation.
diffusion model text-to-image (T2I) Multi-modal Feature Connector attention mechanism pre-trained T2I diffusion model multi-modal prompts personalized images context-aware images