InstructBLIP: Towards General-purpose Vision-Language Models with
  Instruction Tuning
General-purpose language models that can solve various language-domain tasks
have emerged driven by the pre-training and instruction-tuning pipeline.
However, building general-purpose vision-language models is challenging due to
the increased task discrepancy introduced by the additional visual input.
Although vision-language pre-training has been widely studied, vision-language
instruction tuning remains relatively less explored. In this paper, we conduct
a systematic and comprehensive study on vision-language instruction tuning
based on the pre-trained BLIP-2 models. We gather a wide variety of 26 publicly
available datasets, transform them into instruction tuning format and
categorize them into two clusters for held-in instruction tuning and held-out
zero-shot evaluation. Additionally, we introduce instruction-aware visual
feature extraction, a crucial method that enables the model to extract
informative features tailored to the given instruction. The resulting
InstructBLIP models achieve state-of-the-art zero-shot performance across all
13 held-out datasets, substantially outperforming BLIP-2 and the larger
Flamingo. Our models also lead to state-of-the-art performance when finetuned
on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA IMG).
Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over
concurrent multimodal models. All InstructBLIP models have been open-sourced at
https://github.com/salesforce/LAVIS/tree/main/projects/instructblip.
645dafd058f9ee3151487b9c
Wenliang Dai
claimed_verified
2023-05-12T12:56:22.584Z
61bcb994a4d85c3406ad2c23
/avatars/8ff55c0329d3c0986d54af08f8c0c7fa.svg
Wenliang
user
wdaiai
645dafd058f9ee3151487b9d
Junnan Li
claimed_verified
2023-05-12T05:04:29.001Z
61f9d3b54ac99e8a1bae85f4
/avatars/ac47d13204dd22452e4bc46e280842d5.svg
JunnanLi
user
JunnanLi
645dafd058f9ee3151487b9e
Dongxu Li
claimed_verified
2023-05-12T12:28:20.292Z
6220388b354ac333eb998319
/avatars/e04ebeb57f9dc9aa2f6d4732737f6cb8.svg
Dongxu Li
user
dxli
645dafd058f9ee3151487b9f
Anthony Meng Huat Tiong
645dafd058f9ee3151487ba0
Junqi Zhao
645dafd058f9ee3151487ba1
Weisheng Wang
645dafd058f9ee3151487ba2
Boyang Li
645dafd058f9ee3151487ba3
Pascale Fung
645dafd058f9ee3151487ba4
Steven Hoi
InstructBLIP: Towards General-purpose Vision-Language Models with
  Instruction Tuning
General-purpose language models that can solve various language-domain tasks
have emerged driven by the pre-training and instruction-tuning pipeline.
However, building general-purpose vision-language models is challenging due to
the increased task discrepancy introduced by the additional visual input.
Although vision-language pre-training has been widely studied, vision-language
instruction tuning remains relatively less explored. In this paper, we conduct
a systematic and comprehensive study on vision-language instruction tuning
based on the pre-trained BLIP-2 models. We gather a wide variety of 26 publicly
available datasets, transform them into instruction tuning format and
categorize them into two clusters for held-in instruction tuning and held-out
zero-shot evaluation. Additionally, we introduce instruction-aware visual
feature extraction, a crucial method that enables the model to extract
informative features tailored to the given instruction. The resulting
InstructBLIP models achieve state-of-the-art zero-shot performance across all
13 held-out datasets, substantially outperforming BLIP-2 and the larger
Flamingo. Our models also lead to state-of-the-art performance when finetuned
on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA IMG).
Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over
concurrent multimodal models. All InstructBLIP models have been open-sourced at
https://github.com/salesforce/LAVIS/tree/main/projects/instructblip.
InstructBLIP models, built through instruction-aware visual feature extraction on BLIP-2, achieve top performance in vision-language tasks, including zero-shot evaluation and fine-tuned downstream tasks.
vision-language models pre-training instruction-tuning BLIP-2 InstructBLIP instruction-aware visual feature extraction zero-shot performance downstream tasks ScienceQA IMG multimodal models