To See is to Believe: Prompting GPT-4V for Better Visual Instruction
  Tuning
Existing visual instruction tuning methods typically prompt large language
models with textual descriptions to generate instruction-following data.
Despite the promising performance achieved, these descriptions are derived from
image annotations, which are oftentimes coarse-grained. Furthermore, the
instructions might even contradict the visual content without observing the
entire visual context. To address this challenge, we introduce a fine-grained
visual instruction dataset, LVIS-Instruct4V, which contains 220K visually
aligned and context-aware instructions produced by prompting the powerful
GPT-4V with images from LVIS. Through experimental validation and case studies,
we demonstrate that high-quality visual instructional data could improve the
performance of LLaVA-1.5, a state-of-the-art large multimodal model, across a
wide spectrum of benchmarks by clear margins. Notably, by simply replacing the
LLaVA-Instruct with our LVIS-Instruct4V, we achieve better results than LLaVA
on most challenging LMM benchmarks, e.g., LLaVA^w (76.7 vs. 70.7) and MM-Vet
(40.2 vs. 35.4). We release our data and model at
https://github.com/X2FD/LVIS-INSTRUCT4V.
6552f439ecf367738e0e47c8
Junke Wang
admin_assigned
2023-11-14T10:42:50.054Z
642e1e7147833318f0eb3755
/avatars/7ef4f5a099d3eb0fab99b589f33903fa.svg
Junke Wang
user
Daniel0724
6552f439ecf367738e0e47c9
Lingchen Meng
admin_assigned
2023-11-14T10:42:58.013Z
64d852a4bab152b2470bf96e
/avatars/074ef416ea865ca97c208ab2ba55d622.svg
Lingchen Meng
user
menglc
6552f439ecf367738e0e47ca
Zejia Weng
admin_assigned
2023-11-14T10:43:09.163Z
6437cc89961bb61e4639bc47
/avatars/05f5dc29cade3bb5d3146072dadce41c.svg
wengzejia
user
wengzejia
6552f439ecf367738e0e47cb
Bo He
6552f439ecf367738e0e47cc
Zuxuan Wu
6552f439ecf367738e0e47cd
Yu-Gang Jiang
To See is to Believe: Prompting GPT-4V for Better Visual Instruction
  Tuning
Existing visual instruction tuning methods typically prompt large language
models with textual descriptions to generate instruction-following data.
Despite the promising performance achieved, these descriptions are derived from
image annotations, which are oftentimes coarse-grained. Furthermore, the
instructions might even contradict the visual content without observing the
entire visual context. To address this challenge, we introduce a fine-grained
visual instruction dataset, LVIS-Instruct4V, which contains 220K visually
aligned and context-aware instructions produced by prompting the powerful
GPT-4V with images from LVIS. Through experimental validation and case studies,
we demonstrate that high-quality visual instructional data could improve the
performance of LLaVA-1.5, a state-of-the-art large multimodal model, across a
wide spectrum of benchmarks by clear margins. Notably, by simply replacing the
LLaVA-Instruct with our LVIS-Instruct4V, we achieve better results than LLaVA
on most challenging LMM benchmarks, e.g., LLaVA^w (76.7 vs. 70.7) and MM-Vet
(40.2 vs. 35.4). We release our data and model at
https://github.com/X2FD/LVIS-INSTRUCT4V.
A fine-grained visual instruction dataset, LVIS-Instruct4V, improves the performance of LLaVA-1.5 across benchmarks by generating context-aware instructions from images using GPT-4V.
GPT-4V LVIS-Instruct4V LLaVA-1.5 multimodal model LMM benchmarks LLaVA$^w$ MM-Vet