HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into
  Multimodal LLMs at Scale
The rapid development of multimodal large language models (MLLMs), such as
GPT-4V, has led to significant advancements. However, these models still face
challenges in medical multimodal capabilities due to limitations in the
quantity and quality of medical vision-text data, stemming from data privacy
concerns and high annotation costs. While pioneering approaches utilize
PubMed's large-scale, de-identified medical image-text pairs to address these
limitations, they still fall short due to inherent data noise. To tackle this,
we refined medical image-text pairs from PubMed and employed MLLMs (GPT-4V) in
an 'unblinded' capacity to denoise and reformat the data, resulting in the
creation of the PubMedVision dataset with 1.3 million medical VQA samples. Our
validation demonstrates that: (1) PubMedVision can significantly enhance the
medical multimodal capabilities of current MLLMs, showing significant
improvement in benchmarks including the MMMU Health & Medicine track; (2)
manual checks by medical experts and empirical results validate the superior
data quality of our dataset compared to other data construction methods. Using
PubMedVision, we train a 34B medical MLLM HuatuoGPT-Vision, which shows
superior performance in medical multimodal scenarios among open-source MLLMs.
667e437a3a685cd30f9c0f84
Junying Chen
claimed_verified
2024-06-30T20:57:35.571Z
64097dd1b6a334f53e2b3e4c
/avatars/18d036aab5e096054a8706bc78027126.svg
Junying Chen
user
jymcc
667e437a3a685cd30f9c0f85
Ruyi Ouyang
admin_assigned
2024-07-01T07:56:39.424Z
662ccc10e93bb738042a33b0
/avatars/3ae69531fb03e522647e134688459fe7.svg
Ruyi Ouyang
user
OBB1028
667e437a3a685cd30f9c0f86
Anningzhe Gao
claimed_verified
2024-07-25T11:07:34.463Z
66a2271fc56fba8f76bc12ed
/avatars/ef0e9b600d5dac7f160a86fcc08e0b7d.svg
Anningzhe
user
ganz1994
667e437a3a685cd30f9c0f87
Shunian Chen
admin_assigned
2024-07-01T07:56:56.176Z
623be9e1d1eb227788764959
/avatars/b6521b795a59754dbb40123fd4f63b8c.svg
Shunian Chen
user
Shunian
667e437a3a685cd30f9c0f88
Guiming Hardy Chen
667e437a3a685cd30f9c0f89
Xidong Wang
admin_assigned
2024-07-01T07:57:32.221Z
640ed3e9f2d7c41a1e9a9fde
https://cdn-avatars.huggingface.co/v1/production/uploads/1678693486092-640ed3e9f2d7c41a1e9a9fde.jpeg
Xidong Wang
user
Xidong
667e437a3a685cd30f9c0f8a
Ruifei Zhang
admin_assigned
2024-07-01T07:57:38.854Z
6520d42d728c0b6dc7032d7f
/avatars/5acb8548cfbc1a46e0f2c59094e1c30f.svg
Ruifei Zhang
user
ReaFly
667e437a3a685cd30f9c0f8b
Zhenyang Cai
admin_assigned
2024-07-01T07:57:45.400Z
64f1a34f2c5c8b767916447e
https://cdn-avatars.huggingface.co/v1/production/uploads/64f1a34f2c5c8b767916447e/uak2CsMAnxW8q4dwyAOBN.jpeg
Zhenyang Cai
user
Eric3200
667e437a3a685cd30f9c0f8c
Ke Ji
claimed_verified
2024-07-01T09:45:22.865Z
64c24a1b3499e4f3a2fb680c
/avatars/7859c3164071a1797c4d9271c8c6ac2e.svg
Ke Ji
user
kehorizon
667e437a3a685cd30f9c0f8d
Guangjun Yu
667e437a3a685cd30f9c0f8e
Xiang Wan
667e437a3a685cd30f9c0f8f
Benyou Wang
admin_assigned
2024-07-01T07:58:18.395Z
637c6703ca8542a0ba900ccb
/avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
Wang
user
Benyou
HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into
  Multimodal LLMs at Scale
The rapid development of multimodal large language models (MLLMs), such as
GPT-4V, has led to significant advancements. However, these models still face
challenges in medical multimodal capabilities due to limitations in the
quantity and quality of medical vision-text data, stemming from data privacy
concerns and high annotation costs. While pioneering approaches utilize
PubMed's large-scale, de-identified medical image-text pairs to address these
limitations, they still fall short due to inherent data noise. To tackle this,
we refined medical image-text pairs from PubMed and employed MLLMs (GPT-4V) in
an 'unblinded' capacity to denoise and reformat the data, resulting in the
creation of the PubMedVision dataset with 1.3 million medical VQA samples. Our
validation demonstrates that: (1) PubMedVision can significantly enhance the
medical multimodal capabilities of current MLLMs, showing significant
improvement in benchmarks including the MMMU Health & Medicine track; (2)
manual checks by medical experts and empirical results validate the superior
data quality of our dataset compared to other data construction methods. Using
PubMedVision, we train a 34B medical MLLM HuatuoGPT-Vision, which shows
superior performance in medical multimodal scenarios among open-source MLLMs.
The refinement of medical image-text pairs using multimodal large language models enhances medical multimodal capabilities and data quality, leading to superior performance in benchmarks.
multimodal large language models MLLMs GPT-4V PubMed medical image-text pairs denoise reformat PubMedVision MMLMU medical MLLM HuatuoGPT-Vision