LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image
  Understanding
Instruction tuning unlocks the superior capability of Large Language Models
(LLM) to interact with humans. Furthermore, recent instruction-following
datasets include images as visual inputs, collecting responses for image-based
instructions. However, visual instruction-tuned models cannot comprehend
textual details within images well. This work enhances the current visual
instruction tuning pipeline with text-rich images (e.g., movie posters, book
covers, etc.). Specifically, we first use publicly available OCR tools to
collect results on 422K text-rich images from the LAION dataset. Moreover, we
prompt text-only GPT-4 with recognized texts and image captions to generate 16K
conversations, each containing question-answer pairs for text-rich images. By
combining our collected data with previous multi-modal instruction-following
data, our model, LLaVAR, substantially improves the LLaVA model's capability on
text-based VQA datasets (up to 20% accuracy improvement) while achieving an
accuracy of 91.42% on ScienceQA. The GPT-4-based instruction-following
evaluation also demonstrates the improvement of our model on both natural
images and text-rich images. Through qualitative analysis, LLaVAR shows
promising interaction (e.g., reasoning, writing, and elaboration) skills with
humans based on the latest real-world online content that combines text and
images. We make our code/data/models publicly available at
https://llavar.github.io/.
649e2c8500963003b7d23ea6
Yanzhe Zhang
admin_assigned
2023-06-30T19:46:05.998Z
61362307df63bbd1c2459c4c
/avatars/28d3ba4b0fa00acb1eed623ab3620789.svg
Yanzhe Zhang
user
zyanzhe
649e2c8500963003b7d23ea7
Ruiyi Zhang
admin_assigned
2023-07-03T09:46:26.954Z
64668f982da1abc242355cbb
/avatars/c9a248b8d70b1a8f7b78265c98690570.svg
Ruiyi Zhang
user
zhangry868
649e2c8500963003b7d23ea8
Jiuxiang Gu
649e2c8500963003b7d23ea9
Yufan Zhou
admin_assigned
2023-06-30T19:48:11.738Z
635c2c2a7a165601151d3f85
https://cdn-avatars.huggingface.co/v1/production/uploads/1666984965757-noauth.png
Yufan Zhou
user
YfZ
649e2c8500963003b7d23eaa
Nedim Lipka
649e2c8500963003b7d23eab
Diyi Yang
649e2c8500963003b7d23eac
Tong Sun
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image
  Understanding
Instruction tuning unlocks the superior capability of Large Language Models
(LLM) to interact with humans. Furthermore, recent instruction-following
datasets include images as visual inputs, collecting responses for image-based
instructions. However, visual instruction-tuned models cannot comprehend
textual details within images well. This work enhances the current visual
instruction tuning pipeline with text-rich images (e.g., movie posters, book
covers, etc.). Specifically, we first use publicly available OCR tools to
collect results on 422K text-rich images from the LAION dataset. Moreover, we
prompt text-only GPT-4 with recognized texts and image captions to generate 16K
conversations, each containing question-answer pairs for text-rich images. By
combining our collected data with previous multi-modal instruction-following
data, our model, LLaVAR, substantially improves the LLaVA model's capability on
text-based VQA datasets (up to 20% accuracy improvement) while achieving an
accuracy of 91.42% on ScienceQA. The GPT-4-based instruction-following
evaluation also demonstrates the improvement of our model on both natural
images and text-rich images. Through qualitative analysis, LLaVAR shows
promising interaction (e.g., reasoning, writing, and elaboration) skills with
humans based on the latest real-world online content that combines text and
images. We make our code/data/models publicly available at
https://llavar.github.io/.
LLaVAR enhances visual instruction-tuned models by incorporating text recognition from images, resulting in improved performance on text-based VQA datasets and better interaction capabilities.
instruction tuning Large Language Models (LLM) image-based instructions OCR LAION dataset GPT-4 text-only GPT-4 image captions multi-modal instruction-following LLaVAR LLaVA text-based VQA datasets ScienceQA qualitative analysis