Instruct-Imagen: Image Generation with Multi-modal Instruction
This paper presents instruct-imagen, a model that tackles heterogeneous image
generation tasks and generalizes across unseen tasks. We introduce *multi-modal
instruction* for image generation, a task representation articulating a range
of generation intents with precision. It uses natural language to amalgamate
disparate modalities (e.g., text, edge, style, subject, etc.), such that
abundant generation intents can be standardized in a uniform format.
  We then build instruct-imagen by fine-tuning a pre-trained text-to-image
diffusion model with a two-stage framework. First, we adapt the model using the
retrieval-augmented training, to enhance model's capabilities to ground its
generation on external multimodal context. Subsequently, we fine-tune the
adapted model on diverse image generation tasks that requires vision-language
understanding (e.g., subject-driven generation, etc.), each paired with a
multi-modal instruction encapsulating the task's essence. Human evaluation on
various image generation datasets reveals that instruct-imagen matches or
surpasses prior task-specific models in-domain and demonstrates promising
generalization to unseen and more complex tasks.
659772f1ce6e887177e45dfa
Hexiang Hu
claimed_verified
2024-01-12T08:06:43.586Z
643441ccd55dea2d0ec2c309
/avatars/82e99d445e2b513ad7270fa852adbcbb.svg
Hexiang Hu
user
hexianghu
659772f1ce6e887177e45dfb
Kelvin C. K. Chan
659772f1ce6e887177e45dfc
Yu-Chuan Su
659772f1ce6e887177e45dfd
Wenhu Chen
659772f1ce6e887177e45dfe
Yandong Li
659772f1ce6e887177e45dff
Kihyuk Sohn
659772f1ce6e887177e45e00
Yang Zhao
claimed_verified
2024-01-08T08:48:00.827Z
631ba6aaf6bc4be4a653b2c7
/avatars/c026f47a042711127d3b8431f6217f55.svg
Yang Zhao
user
YangNaruto
659772f1ce6e887177e45e01
Xue Ben
659772f1ce6e887177e45e02
Boqing Gong
659772f1ce6e887177e45e03
William Cohen
659772f1ce6e887177e45e04
Ming-Wei Chang
659772f1ce6e887177e45e05
Xuhui Jia
Instruct-Imagen: Image Generation with Multi-modal Instruction
This paper presents instruct-imagen, a model that tackles heterogeneous image
generation tasks and generalizes across unseen tasks. We introduce *multi-modal
instruction* for image generation, a task representation articulating a range
of generation intents with precision. It uses natural language to amalgamate
disparate modalities (e.g., text, edge, style, subject, etc.), such that
abundant generation intents can be standardized in a uniform format.
  We then build instruct-imagen by fine-tuning a pre-trained text-to-image
diffusion model with a two-stage framework. First, we adapt the model using the
retrieval-augmented training, to enhance model's capabilities to ground its
generation on external multimodal context. Subsequently, we fine-tune the
adapted model on diverse image generation tasks that requires vision-language
understanding (e.g., subject-driven generation, etc.), each paired with a
multi-modal instruction encapsulating the task's essence. Human evaluation on
various image generation datasets reveals that instruct-imagen matches or
surpasses prior task-specific models in-domain and demonstrates promising
generalization to unseen and more complex tasks.
Instruct-Imagen achieves robust and generalized image generation through fine-tuning a pre-trained text-to-image diffusion model with multi-modal instructions and retrieval-augmented training.
multi-modal instruction text-to-image diffusion model retrieval-augmented training subject-driven generation vision-language understanding