WorldMedQA-V: a multilingual, multimodal medical examination dataset for
  multimodal language models evaluation
Multimodal/vision language models (VLMs) are increasingly being deployed in
healthcare settings worldwide, necessitating robust benchmarks to ensure their
safety, efficacy, and fairness. Multiple-choice question and answer (QA)
datasets derived from national medical examinations have long served as
valuable evaluation tools, but existing datasets are largely text-only and
available in a limited subset of languages and countries. To address these
challenges, we present WorldMedQA-V, an updated multilingual, multimodal
benchmarking dataset designed to evaluate VLMs in healthcare. WorldMedQA-V
includes 568 labeled multiple-choice QAs paired with 568 medical images from
four countries (Brazil, Israel, Japan, and Spain), covering original languages
and validated English translations by native clinicians, respectively. Baseline
performance for common open- and closed-source models are provided in the local
language and English translations, and with and without images provided to the
model. The WorldMedQA-V benchmark aims to better match AI systems to the
diverse healthcare environments in which they are deployed, fostering more
equitable, effective, and representative applications.
6710cf9452f304c3e265e2ab
João Matos
claimed_verified
2024-10-18T08:14:18.443Z
6668600d9a0868622619157f
https://cdn-avatars.huggingface.co/v1/production/uploads/6668600d9a0868622619157f/Y7Avc94fxlfSiQSweUqvo.jpeg
João matos
user
joamats
6710cf9452f304c3e265e2ac
Shan Chen
claimed_verified
2024-10-17T11:23:46.339Z
63600b93d9e4214b9a67807c
https://cdn-avatars.huggingface.co/v1/production/uploads/63600b93d9e4214b9a67807c/x_VE45ZPoaUFiryYl_5J5.png
Shan Chen
user
shanchen
6710cf9452f304c3e265e2ad
Siena Placino
6710cf9452f304c3e265e2ae
Yingya Li
6710cf9452f304c3e265e2af
Juan Carlos Climent Pardo
6710cf9452f304c3e265e2b0
Daphna Idan
6710cf9452f304c3e265e2b1
Takeshi Tohyama
6710cf9452f304c3e265e2b2
David Restrepo
6710cf9452f304c3e265e2b3
Luis F. Nakayama
claimed_verified
2025-04-29T12:13:34.468Z
629cec7ebf5b16f052645be8
/avatars/0a40a87ef88a457d14bb38090f6a615e.svg
Luis Filipe Nakayama
user
luisnaka
6710cf9452f304c3e265e2b4
Jose M. M. Pascual-Leone
6710cf9452f304c3e265e2b5
Guergana Savova
6710cf9452f304c3e265e2b6
Hugo Aerts
6710cf9452f304c3e265e2b7
Leo A. Celi
6710cf9452f304c3e265e2b8
A. Ian Wong
6710cf9452f304c3e265e2b9
Danielle S. Bitterman
6710cf9452f304c3e265e2ba
Jack Gallifant
WorldMedQA-V: a multilingual, multimodal medical examination dataset for
  multimodal language models evaluation
Multimodal/vision language models (VLMs) are increasingly being deployed in
healthcare settings worldwide, necessitating robust benchmarks to ensure their
safety, efficacy, and fairness. Multiple-choice question and answer (QA)
datasets derived from national medical examinations have long served as
valuable evaluation tools, but existing datasets are largely text-only and
available in a limited subset of languages and countries. To address these
challenges, we present WorldMedQA-V, an updated multilingual, multimodal
benchmarking dataset designed to evaluate VLMs in healthcare. WorldMedQA-V
includes 568 labeled multiple-choice QAs paired with 568 medical images from
four countries (Brazil, Israel, Japan, and Spain), covering original languages
and validated English translations by native clinicians, respectively. Baseline
performance for common open- and closed-source models are provided in the local
language and English translations, and with and without images provided to the
model. The WorldMedQA-V benchmark aims to better match AI systems to the
diverse healthcare environments in which they are deployed, fostering more
equitable, effective, and representative applications.
WorldMedQA-V is a multilingual, multimodal dataset for evaluating vision language models in healthcare, incorporating images and questions from multiple countries and languages.
vision language models multimodal benchmarks multiple-choice question and answer medical images