MUMU: Bootstrapping Multimodal Image Generation from Text-to-Image Data
We train a model to generate images from multimodal prompts of interleaved
text and images such as "a <picture of a man> man and his <picture of a dog>
dog in an <picture of a cartoon> animated style." We bootstrap a multimodal
dataset by extracting semantically meaningful image crops corresponding to
words in the image captions of synthetically generated and publicly available
text-image data. Our model, MUMU, is composed of a vision-language model
encoder with a diffusion decoder and is trained on a single 8xH100 GPU node.
Despite being only trained on crops from the same image, MUMU learns to compose
inputs from different images into a coherent output. For example, an input of a
realistic person and a cartoon will output the same person in the cartoon
style, and an input of a standing subject and a scooter will output the subject
riding the scooter. As a result, our model generalizes to tasks such as style
transfer and character consistency. Our results show the promise of using
multimodal models as general purpose controllers for image generation.
667ec70aa7d8b1deba80afa8
William Berman
admin_assigned
2024-06-28T14:57:47.457Z
63407fadb78ed99eab00203d
https://cdn-avatars.huggingface.co/v1/production/uploads/1668468846504-63407fadb78ed99eab00203d.jpeg
Will Berman
user
williamberman
667ec70aa7d8b1deba80afa9
Alexander Peysakhovich
MUMU: Bootstrapping Multimodal Image Generation from Text-to-Image Data
We train a model to generate images from multimodal prompts of interleaved
text and images such as "a <picture of a man> man and his <picture of a dog>
dog in an <picture of a cartoon> animated style." We bootstrap a multimodal
dataset by extracting semantically meaningful image crops corresponding to
words in the image captions of synthetically generated and publicly available
text-image data. Our model, MUMU, is composed of a vision-language model
encoder with a diffusion decoder and is trained on a single 8xH100 GPU node.
Despite being only trained on crops from the same image, MUMU learns to compose
inputs from different images into a coherent output. For example, an input of a
realistic person and a cartoon will output the same person in the cartoon
style, and an input of a standing subject and a scooter will output the subject
riding the scooter. As a result, our model generalizes to tasks such as style
transfer and character consistency. Our results show the promise of using
multimodal models as general purpose controllers for image generation.
MUMU, a multimodal model combining vision-language encoding and diffusion decoding, synthesizes coherent images from text-image prompts and generalizes to tasks like style transfer and character consistency.
multimodal prompts vision-language model diffusion decoder semantic meaningful image crops multimodal dataset style transfer character consistency general purpose controllers