MultiModal-GPT: A Vision and Language Model for Dialogue with Humans
We present a vision and language model named MultiModal-GPT to conduct
multi-round dialogue with humans. MultiModal-GPT can follow various
instructions from humans, such as generating a detailed caption, counting the
number of interested objects, and answering general questions from users.
MultiModal-GPT is parameter-efficiently fine-tuned from OpenFlamingo, with
Low-rank Adapter (LoRA) added both in the cross-attention part and the
self-attention part of the language model. We first construct instruction
templates with vision and language data for multi-modality instruction tuning
to make the model understand and follow human instructions. We find the quality
of training data is vital for the dialogue performance, where few data
containing short answers can lead the model to respond shortly to any
instructions. To further enhance the ability to chat with humans of the
MultiModal-GPT, we utilize language-only instruction-following data to train
the MultiModal-GPT jointly. The joint training of language-only and
visual-language instructions with the same instruction template
effectively improves dialogue performance. Various demos show the ability of
continuous dialogue of MultiModal-GPT with humans. Code and demo are at
https://github.com/open-mmlab/Multimodal-GPT
6459a91fabdbb77c4c6d9990
Tao Gong
6459a91fabdbb77c4c6d9991
Chengqi Lyu
6459a91fabdbb77c4c6d9992
Shilong Zhang
admin_assigned
2023-06-02T14:08:09.911Z
6424ffce46d202ad3d918a67
https://cdn-avatars.huggingface.co/v1/production/uploads/6424ffce46d202ad3d918a67/gmYmOA072fP_5cJLc9Qs4.jpeg
Shilong Zhang
user
shilongz
6459a91fabdbb77c4c6d9993
Yudong Wang
claimed_verified
2025-05-09T07:24:12.126Z
63be286fb3b8c44f8cecc16f
https://cdn-avatars.huggingface.co/v1/production/uploads/63be286fb3b8c44f8cecc16f/1CIkfEKoTnBYdYDSuQ8AT.jpeg
Yudong Wang
user
BigDong
6459a91fabdbb77c4c6d9994
Miao Zheng
6459a91fabdbb77c4c6d9995
Qian Zhao
6459a91fabdbb77c4c6d9996
Kuikun Liu
claimed_verified
2024-08-02T07:35:36.973Z
63fd691794cc8f815d50c112
/avatars/87305d1cbfcc717e910ccdfaf0568f80.svg
liu
user
Harold-lkk
6459a91fabdbb77c4c6d9997
Wenwei Zhang
claimed_verified
2025-02-11T10:36:50.932Z
64e8505321540e1da3226b54
/avatars/18958b8406d1ce492b54c1c839f18c54.svg
Wenwei Zhang
user
ZwwWayne
6459a91fabdbb77c4c6d9998
Ping Luo
6459a91fabdbb77c4c6d9999
Kai Chen
MultiModal-GPT: A Vision and Language Model for Dialogue with Humans
We present a vision and language model named MultiModal-GPT to conduct
multi-round dialogue with humans. MultiModal-GPT can follow various
instructions from humans, such as generating a detailed caption, counting the
number of interested objects, and answering general questions from users.
MultiModal-GPT is parameter-efficiently fine-tuned from OpenFlamingo, with
Low-rank Adapter (LoRA) added both in the cross-attention part and the
self-attention part of the language model. We first construct instruction
templates with vision and language data for multi-modality instruction tuning
to make the model understand and follow human instructions. We find the quality
of training data is vital for the dialogue performance, where few data
containing short answers can lead the model to respond shortly to any
instructions. To further enhance the ability to chat with humans of the
MultiModal-GPT, we utilize language-only instruction-following data to train
the MultiModal-GPT jointly. The joint training of language-only and
visual-language instructions with the same instruction template
effectively improves dialogue performance. Various demos show the ability of
continuous dialogue of MultiModal-GPT with humans. Code and demo are at
https://github.com/open-mmlab/Multimodal-GPT
MultiModal-GPT, fine-tuned with Low-rank Adapter, enhances dialogue quality by using mixed instruction templates from language-only and visual-language data.
parameter-efficient fine-tuning Low-rank Adapter LoRA cross-attention self-attention instruction templates multi-modality instruction tuning dialogue performance continuous dialogue