MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular
  Annotations for Medicine
This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal
dataset for medicine, covering over 25 million images across 10 modalities,
with multigranular annotations for more than 65 diseases. These enriched
annotations encompass both global textual information, such as disease/lesion
type, modality, region-specific descriptions, and inter-regional relationships,
as well as detailed local annotations for regions of interest (ROIs), including
bounding boxes, segmentation masks. Unlike existing approach which is limited
by the availability of image-text pairs, we have developed the first automated
pipeline that scales up multimodal data by generating multigranular visual and
texual annotations (in the form of image-ROI-description triplets) without the
need for any paired text descriptions. Specifically, data from over 90
different sources have been collected, preprocessed, and grounded using
domain-specific expert models to identify ROIs related to abnormal regions. We
then build a comprehensive knowledge base and prompt multimodal large language
models to perform retrieval-augmented generation with the identified ROIs as
guidance, resulting in multigranular texual descriptions. Compared to existing
datasets, MedTrinity-25M provides the most enriched annotations, supporting a
comprehensive range of multimodal tasks such as captioning and report
generation, as well as vision-centric tasks like classification and
segmentation. Pretraining on MedTrinity-25M, our model achieves
state-of-the-art performance on VQA-RAD and PathVQA, surpassing both multimodal
large language models and other representative SoTA approaches. This dataset
can also be utilized to support large-scale pre-training of multimodal medical
AI models, contributing to the development of future foundation models in the
medical domain.
66b2d7e8fa7069c7f1f0887b
Yunfei Xie
admin_assigned
2024-08-08T12:20:04.588Z
65f6afa36ea40b9a29dd45ca
/avatars/a80a25fa70a89a3bf3d50ea2866f0df1.svg
Yunfei Xie
user
yunfeixie
66b2d7e8fa7069c7f1f0887c
Ce Zhou
66b2d7e8fa7069c7f1f0887d
Lang Gao
66b2d7e8fa7069c7f1f0887e
Juncheng Wu
admin_assigned
2024-08-08T12:18:59.704Z
641a58d791e3376a0579f011
/avatars/bb4745bb83ac0ee094b43e1e5562d743.svg
Juncheng Wu
user
JunchengWu
66b2d7e8fa7069c7f1f0887f
Xianhang Li
admin_assigned
2024-08-08T12:18:51.477Z
645e72d4b5c9a8666d0ddc1c
/avatars/f0f1684731eb5ae148eadf841cceb18e.svg
xhl
user
Xianhang
66b2d7e8fa7069c7f1f08880
Hong-Yu Zhou
admin_assigned
2024-08-08T12:18:41.441Z
63983ea1ec07edfe479b9967
/avatars/2e4fcdac836beb1829bfea335f285fd1.svg
Hong-Yu Zhou
user
hyzhou
66b2d7e8fa7069c7f1f08881
Sheng Liu
admin_assigned
2024-08-08T12:18:35.018Z
653585666141b3927a083b4f
https://cdn-avatars.huggingface.co/v1/production/uploads/653585666141b3927a083b4f/EpMnW_gKz9FV3OvogvUzi.jpeg
Sheng Liu
user
shengliu66
66b2d7e8fa7069c7f1f08882
Lei Xing
66b2d7e8fa7069c7f1f08883
James Zou
admin_assigned
2024-08-08T12:18:01.076Z
648a769003fc4a3938bb7943
/avatars/7647f99abdcca4251fcac7783b6fcc8d.svg
zou
user
jameszou707
66b2d7e8fa7069c7f1f08884
Cihang Xie
admin_assigned
2024-08-08T12:17:55.313Z
645eb61da3c5cd8a16efffff
/avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
Cihang Xie
user
cihangxie
66b2d7e8fa7069c7f1f08885
Yuyin Zhou
MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular
  Annotations for Medicine
This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal
dataset for medicine, covering over 25 million images across 10 modalities,
with multigranular annotations for more than 65 diseases. These enriched
annotations encompass both global textual information, such as disease/lesion
type, modality, region-specific descriptions, and inter-regional relationships,
as well as detailed local annotations for regions of interest (ROIs), including
bounding boxes, segmentation masks. Unlike existing approach which is limited
by the availability of image-text pairs, we have developed the first automated
pipeline that scales up multimodal data by generating multigranular visual and
texual annotations (in the form of image-ROI-description triplets) without the
need for any paired text descriptions. Specifically, data from over 90
different sources have been collected, preprocessed, and grounded using
domain-specific expert models to identify ROIs related to abnormal regions. We
then build a comprehensive knowledge base and prompt multimodal large language
models to perform retrieval-augmented generation with the identified ROIs as
guidance, resulting in multigranular texual descriptions. Compared to existing
datasets, MedTrinity-25M provides the most enriched annotations, supporting a
comprehensive range of multimodal tasks such as captioning and report
generation, as well as vision-centric tasks like classification and
segmentation. Pretraining on MedTrinity-25M, our model achieves
state-of-the-art performance on VQA-RAD and PathVQA, surpassing both multimodal
large language models and other representative SoTA approaches. This dataset
can also be utilized to support large-scale pre-training of multimodal medical
AI models, contributing to the development of future foundation models in the
medical domain.
MedTrinity-25M, a large-scale multimodal medical dataset with extensive annotations, enables automated generation of detailed visual and textual descriptions, achieving state-of-the-art performance in tasks like VQA-RAD and PathVQA.
multimodal dataset image-ROI-description triplets domain-specific expert models retrieval-augmented generation multigranular annotations bounding boxes segmentation masks pretraining VQA-RAD PathVQA