VL-GPT: A Generative Pre-trained Transformer for Vision and Language
  Understanding and Generation
In this work, we introduce Vision-Language Generative Pre-trained Transformer
(VL-GPT), a transformer model proficient at concurrently perceiving and
generating visual and linguistic data. VL-GPT achieves a unified pre-training
approach for both image and text modalities by employing a straightforward
auto-regressive objective, thereby enabling the model to process image and text
as seamlessly as a language model processes text. To accomplish this, we
initially propose a novel image tokenizer-detokenizer framework for visual
data, specifically designed to transform raw images into a sequence of
continuous embeddings and reconstruct them accordingly. In combination with the
existing text tokenizer and detokenizer, this framework allows for the encoding
of interleaved image-text data into a multimodal sequence, which can
subsequently be fed into the transformer model. Consequently, VL-GPT can
perform large-scale pre-training on multimodal corpora utilizing a unified
auto-regressive objective (i.e., next-token prediction). Upon completion of
pre-training, VL-GPT exhibits remarkable zero-shot and few-shot performance
across a diverse range of vision and language understanding and generation
tasks, including image captioning, visual question answering, text-to-image
generation, and more. Additionally, the pre-trained model retrains in-context
learning capabilities when provided with multimodal prompts. We further conduct
instruction tuning on our VL-GPT, highlighting its exceptional potential for
multimodal assistance. The source code and model weights shall be released.
657bd0137e310c370218ae25
Jinguo Zhu
657bd0137e310c370218ae26
Xiaohan Ding
admin_assigned
2023-12-15T13:19:07.753Z
6564f049957ac688429a6754
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/nE6q5TlyLKlJpbxCKoDN8.png
xiaohanding
user
DingXiaoH
657bd0137e310c370218ae27
Yixiao Ge
admin_assigned
2023-12-15T13:19:16.086Z
640e9762b03f4cd29f58d982
/avatars/81da37d628163fe3e094b247c7c3a3b5.svg
Yixiao Ge
user
yxgeee
657bd0137e310c370218ae28
Yuying Ge
657bd0137e310c370218ae29
Sijie Zhao
admin_assigned
2023-12-15T13:19:25.529Z
652cf6bad78452c474434fa6
/avatars/106bd7a6f8af3ae3b079a0533f6fb619.svg
Sijie Zhao
user
walkingshadow
657bd0137e310c370218ae2a
Hengshuang Zhao
657bd0137e310c370218ae2b
Xiaohua Wang
admin_assigned
2023-12-15T13:19:43.538Z
642d7a9e355bb9a2a436626d
/avatars/893c8ec0442d451cb493d2b1d0e2d5f3.svg
XiaoHua Wang
user
myARmy10
657bd0137e310c370218ae2c
Ying Shan
VL-GPT: A Generative Pre-trained Transformer for Vision and Language
  Understanding and Generation
In this work, we introduce Vision-Language Generative Pre-trained Transformer
(VL-GPT), a transformer model proficient at concurrently perceiving and
generating visual and linguistic data. VL-GPT achieves a unified pre-training
approach for both image and text modalities by employing a straightforward
auto-regressive objective, thereby enabling the model to process image and text
as seamlessly as a language model processes text. To accomplish this, we
initially propose a novel image tokenizer-detokenizer framework for visual
data, specifically designed to transform raw images into a sequence of
continuous embeddings and reconstruct them accordingly. In combination with the
existing text tokenizer and detokenizer, this framework allows for the encoding
of interleaved image-text data into a multimodal sequence, which can
subsequently be fed into the transformer model. Consequently, VL-GPT can
perform large-scale pre-training on multimodal corpora utilizing a unified
auto-regressive objective (i.e., next-token prediction). Upon completion of
pre-training, VL-GPT exhibits remarkable zero-shot and few-shot performance
across a diverse range of vision and language understanding and generation
tasks, including image captioning, visual question answering, text-to-image
generation, and more. Additionally, the pre-trained model retrains in-context
learning capabilities when provided with multimodal prompts. We further conduct
instruction tuning on our VL-GPT, highlighting its exceptional potential for
multimodal assistance. The source code and model weights shall be released.
VL-GPT, a transformer model, unifies pre-training for images and text using a novel tokenizer, enabling zero-shot and few-shot performance across various vision and language tasks.
transformer model Vision-Language Generative Pre-trained Transformer VL-GPT auto-regressive objective image tokenizer-detokenizer continuous embeddings multimodal sequence interleaved image-text data zero-shot performance few-shot performance image captioning visual question answering text-to-image generation retrain in-context learning instruction tuning