Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation
  with Multimodal Generative Pretraining
We present Lumina-mGPT, a family of multimodal autoregressive models capable
of various vision and language tasks, particularly excelling in generating
flexible photorealistic images from text descriptions. Unlike existing
autoregressive image generation approaches, Lumina-mGPT employs a pretrained
decoder-only transformer as a unified framework for modeling multimodal token
sequences. Our key insight is that a simple decoder-only transformer with
multimodal Generative PreTraining (mGPT), utilizing the next-token prediction
objective on massive interleaved text-image sequences, can learn broad and
general multimodal capabilities, thereby illuminating photorealistic
text-to-image generation. Building on these pretrained models, we propose
Flexible Progressive Supervised Finetuning (FP-SFT) on high-quality image-text
pairs to fully unlock their potential for high-aesthetic image synthesis at any
resolution while maintaining their general multimodal capabilities.
Furthermore, we introduce Ominiponent Supervised Finetuning (Omni-SFT),
transforming Lumina-mGPT into a foundation model that seamlessly achieves
omnipotent task unification. The resulting model demonstrates versatile
multimodal capabilities, including visual generation tasks like flexible
text-to-image generation and controllable generation, visual recognition tasks
like segmentation and depth estimation, and vision-language tasks like
multiturn visual question answering. Additionally, we analyze the differences
and similarities between diffusion-based and autoregressive methods in a direct
comparison.
66b19b1cd0037f5b2d5016e7
Dongyang Liu
admin_assigned
2024-08-06T15:23:53.143Z
646f1bef075e11ca78da3bb7
https://cdn-avatars.huggingface.co/v1/production/uploads/646f1bef075e11ca78da3bb7/gNS-ikyZXYeMrf4a7HTQE.jpeg
Dongyang Liu (Chris Liu)
user
Cxxs
66b19b1cd0037f5b2d5016e8
Shitian Zhao
claimed_verified
2024-08-06T09:59:30.180Z
62c66504031996c36c86976a
https://cdn-avatars.huggingface.co/v1/production/uploads/62c66504031996c36c86976a/wIq0YJhkWnEhlzsh-TGYO.png
steve z
user
stzhao
66b19b1cd0037f5b2d5016e9
Le Zhuo
admin_assigned
2024-08-06T09:27:39.499Z
6358a167f56b03ec9147074d
/avatars/e54ea7bf0c240cf76d538296efb3976c.svg
Le Zhuo
user
JackyZhuo
66b19b1cd0037f5b2d5016ea
Weifeng Lin
admin_assigned
2024-08-06T09:28:32.532Z
63483f3450fe0799927b9b0f
/avatars/66c2be2d03dafcb91cec0239ca7b8b44.svg
Weifeng Lin
user
Singaless
66b19b1cd0037f5b2d5016eb
Yu Qiao
66b19b1cd0037f5b2d5016ec
Hongsheng Li
admin_assigned
2024-08-06T09:28:51.119Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
66b19b1cd0037f5b2d5016ed
Peng Gao
Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation
  with Multimodal Generative Pretraining
We present Lumina-mGPT, a family of multimodal autoregressive models capable
of various vision and language tasks, particularly excelling in generating
flexible photorealistic images from text descriptions. Unlike existing
autoregressive image generation approaches, Lumina-mGPT employs a pretrained
decoder-only transformer as a unified framework for modeling multimodal token
sequences. Our key insight is that a simple decoder-only transformer with
multimodal Generative PreTraining (mGPT), utilizing the next-token prediction
objective on massive interleaved text-image sequences, can learn broad and
general multimodal capabilities, thereby illuminating photorealistic
text-to-image generation. Building on these pretrained models, we propose
Flexible Progressive Supervised Finetuning (FP-SFT) on high-quality image-text
pairs to fully unlock their potential for high-aesthetic image synthesis at any
resolution while maintaining their general multimodal capabilities.
Furthermore, we introduce Ominiponent Supervised Finetuning (Omni-SFT),
transforming Lumina-mGPT into a foundation model that seamlessly achieves
omnipotent task unification. The resulting model demonstrates versatile
multimodal capabilities, including visual generation tasks like flexible
text-to-image generation and controllable generation, visual recognition tasks
like segmentation and depth estimation, and vision-language tasks like
multiturn visual question answering. Additionally, we analyze the differences
and similarities between diffusion-based and autoregressive methods in a direct
comparison.
Lumina-mGPT, a family of pretrained decoder-only transformer models, excels in various vision and language tasks, particularly text-to-image generation, through Flexible Progressive Supervised Finetuning and Ominiponent Supervised Finetuning.
multimodal autoregressive models photorealistic images pretrained decoder-only transformer multimodal Generative PreTraining (mGPT) next-token prediction Flexible Progressive Supervised Finetuning (FP-SFT) Ominiponent Supervised Finetuning (Omni-SFT) high-quality image-text pairs high-aesthetic image synthesis task unification visual generation tasks visual recognition tasks vision-language tasks diffusion-based methods autoregressive methods