LLaVA-Med: Training a Large Language-and-Vision Assistant for
  Biomedicine in One Day
Conversational generative AI has demonstrated remarkable promise for
empowering biomedical practitioners, but current investigations focus on
unimodal text. Multimodal conversational AI has seen rapid progress by
leveraging billions of image-text pairs from the public web, but such
general-domain vision-language models still lack sophistication in
understanding and conversing about biomedical images. In this paper, we propose
a cost-efficient approach for training a vision-language conversational
assistant that can answer open-ended research questions of biomedical images.
The key idea is to leverage a large-scale, broad-coverage biomedical
figure-caption dataset extracted from PubMed Central, use GPT-4 to
self-instruct open-ended instruction-following data from the captions, and then
fine-tune a large general-domain vision-language model using a novel curriculum
learning method. Specifically, the model first learns to align biomedical
vocabulary using the figure-caption pairs as is, then learns to master
open-ended conversational semantics using GPT-4 generated instruction-following
data, broadly mimicking how a layperson gradually acquires biomedical
knowledge. This enables us to train a Large Language and Vision Assistant for
BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med
exhibits excellent multimodal conversational capability and can follow
open-ended instruction to assist with inquiries about a biomedical image. On
three standard biomedical visual question answering datasets, LLaVA-Med
outperforms previous supervised state-of-the-art on certain metrics. To
facilitate biomedical multimodal research, we will release our
instruction-following data and the LLaVA-Med model.
647940a83b7f8b1f624c07ec
Chunyuan Li
claimed_verified
2023-06-02T07:16:44.404Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
647940a83b7f8b1f624c07ed
Cliff Wong
647940a83b7f8b1f624c07ee
Sheng Zhang
claimed_verified
2023-06-05T10:07:32.281Z
6234c11b7d5de9839bc44163
/avatars/3ca569596f9c7134e8d4b560a06ee1e7.svg
Sheng Zhang
user
shengz
647940a83b7f8b1f624c07ef
Naoto Usuyama
claimed_verified
2023-06-05T10:06:32.327Z
5f243561e923d665e616263f
https://cdn-avatars.huggingface.co/v1/production/uploads/1596208460315-noauth.jpeg
Naoto Usuyama
user
naotous
647940a83b7f8b1f624c07f0
Haotian Liu
claimed_verified
2023-06-05T10:07:23.514Z
63898b61ec1f539adc0f4da2
https://cdn-avatars.huggingface.co/v1/production/uploads/1674248167280-63898b61ec1f539adc0f4da2.jpeg
Haotian Liu
user
liuhaotian
647940a83b7f8b1f624c07f1
Jianwei Yang
647940a83b7f8b1f624c07f2
Tristan Naumann
admin_assigned
2023-06-05T08:55:11.048Z
5e5870466bc35159a08ca572
https://cdn-avatars.huggingface.co/v1/production/uploads/5e5870466bc35159a08ca572/pT6gEEs8RLRJGeM-faNWj.jpeg
Tristan Naumann
user
tnaumann
647940a83b7f8b1f624c07f3
Hoifung Poon
647940a83b7f8b1f624c07f4
Jianfeng Gao
LLaVA-Med: Training a Large Language-and-Vision Assistant for
  Biomedicine in One Day
Conversational generative AI has demonstrated remarkable promise for
empowering biomedical practitioners, but current investigations focus on
unimodal text. Multimodal conversational AI has seen rapid progress by
leveraging billions of image-text pairs from the public web, but such
general-domain vision-language models still lack sophistication in
understanding and conversing about biomedical images. In this paper, we propose
a cost-efficient approach for training a vision-language conversational
assistant that can answer open-ended research questions of biomedical images.
The key idea is to leverage a large-scale, broad-coverage biomedical
figure-caption dataset extracted from PubMed Central, use GPT-4 to
self-instruct open-ended instruction-following data from the captions, and then
fine-tune a large general-domain vision-language model using a novel curriculum
learning method. Specifically, the model first learns to align biomedical
vocabulary using the figure-caption pairs as is, then learns to master
open-ended conversational semantics using GPT-4 generated instruction-following
data, broadly mimicking how a layperson gradually acquires biomedical
knowledge. This enables us to train a Large Language and Vision Assistant for
BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med
exhibits excellent multimodal conversational capability and can follow
open-ended instruction to assist with inquiries about a biomedical image. On
three standard biomedical visual question answering datasets, LLaVA-Med
outperforms previous supervised state-of-the-art on certain metrics. To
facilitate biomedical multimodal research, we will release our
instruction-following data and the LLaVA-Med model.
A cost-efficient approach using a biomedical figure-caption dataset and GPT-4-generated instruction-following data trains a vision-language conversational assistant for biomedical images, achieving state-of-the-art performance on visual question answering tasks.
vision-language models biomedical images multimodal conversational AI GPT-4 curriculum learning biomedical figure-caption dataset instruction-following data Large Language and Vision Assistant for BioMedicine (LLaVA-Med) biomedical visual question answering datasets