Unimedvl: Unifying Medical Multimodal Understanding And Generation
  Through Observation-Knowledge-Analysis
Medical diagnostic applications require models that can process multimodal
medical inputs (images, patient histories, lab results) and generate diverse
outputs including both textual reports and visual content (annotations,
segmentation masks, and images). Despite this need, existing medical AI systems
disrupt this unified process: medical image understanding models interpret
images but cannot generate visual outputs, while medical image generation
models synthesize images but cannot provide textual explanations. This leads to
gaps in data representation, feature integration, and task-level multimodal
capabilities. To this end, we propose a multi-level framework that draws
inspiration from diagnostic workflows through the
Observation-Knowledge-Analysis (OKA) paradigm. Specifically, at the observation
level, we construct UniMed-5M, a dataset comprising over 5.6M samples that
reformat diverse unimodal data into multimodal pairs for foundational
observation. At the knowledge level, we propose Progressive Curriculum Learning
that systematically introduces medical multimodal knowledge. At the analysis
level, we introduce UniMedVL, the first medical unified multimodal model for
the simultaneous analysis of image understanding and generation tasks within a
single architecture. UniMedVL achieves superior performance on five medical
image understanding benchmarks, while matching specialized models in generation
quality across eight medical imaging modalities. Crucially, our unified
architecture enables bidirectional knowledge sharing: generation tasks enhance
visual understanding features, demonstrating that integrating traditionally
separate capabilities within a single medical framework unlocks improvements
across diverse medical vision-language tasks. Code is available at
https://github.com/uni-medical/UniMedVL.
68f5cd4d8589920bf4d321cc
Junzhi Ning
claimed_verified
2025-10-22T15:46:18.243Z
67193c60f89011dc142caf07
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/HC3fjL9V0hhpmjc4kWnGF.png
Junzhi Ning
user
junzhin
68f5cd4d8589920bf4d321cd
Wei Li
claimed_verified
2025-10-23T15:01:24.865Z
65439f54d0f52498c6bf412c
/avatars/666b722a685564b8e10f69e6b68e5504.svg
liwei
user
marcry
68f5cd4d8589920bf4d321ce
Cheng Tang
68f5cd4d8589920bf4d321cf
Jiashi Lin
68f5cd4d8589920bf4d321d0
Chenglong Ma
68f5cd4d8589920bf4d321d1
Chaoyang Zhang
68f5cd4d8589920bf4d321d2
Jiyao Liu
68f5cd4d8589920bf4d321d3
Ying Chen
68f5cd4d8589920bf4d321d4
Shujian Gao
68f5cd4d8589920bf4d321d5
Lihao Liu
68f5cd4d8589920bf4d321d6
Yuandong Pu
claimed_verified
2025-10-21T14:04:45.300Z
625d5b9f0bec31f086e04cd9
https://cdn-avatars.huggingface.co/v1/production/uploads/1650285458447-noauth.jpeg
YuandongPu
user
Andrew613
68f5cd4d8589920bf4d321d7
Huihui Xu
68f5cd4d8589920bf4d321d8
Chenhui Gou
68f5cd4d8589920bf4d321d9
Ziyan Huang
68f5cd4d8589920bf4d321da
Yi Xin
68f5cd4d8589920bf4d321db
Qi Qin
68f5cd4d8589920bf4d321dc
Zhongying Deng
68f5cd4d8589920bf4d321dd
Diping Song
68f5cd4d8589920bf4d321de
Bin Fu
68f5cd4d8589920bf4d321df
Guang Yang
68f5cd4d8589920bf4d321e0
Yuanfeng Ji
68f5cd4d8589920bf4d321e1
Tianbin Li
68f5cd4d8589920bf4d321e2
Yanzhou Su
68f5cd4d8589920bf4d321e3
Jin Ye
68f5cd4d8589920bf4d321e4
Shixiang Tang
68f5cd4d8589920bf4d321e5
Ming Hu
68f5cd4d8589920bf4d321e6
Junjun He
Unimedvl: Unifying Medical Multimodal Understanding And Generation
  Through Observation-Knowledge-Analysis
Medical diagnostic applications require models that can process multimodal
medical inputs (images, patient histories, lab results) and generate diverse
outputs including both textual reports and visual content (annotations,
segmentation masks, and images). Despite this need, existing medical AI systems
disrupt this unified process: medical image understanding models interpret
images but cannot generate visual outputs, while medical image generation
models synthesize images but cannot provide textual explanations. This leads to
gaps in data representation, feature integration, and task-level multimodal
capabilities. To this end, we propose a multi-level framework that draws
inspiration from diagnostic workflows through the
Observation-Knowledge-Analysis (OKA) paradigm. Specifically, at the observation
level, we construct UniMed-5M, a dataset comprising over 5.6M samples that
reformat diverse unimodal data into multimodal pairs for foundational
observation. At the knowledge level, we propose Progressive Curriculum Learning
that systematically introduces medical multimodal knowledge. At the analysis
level, we introduce UniMedVL, the first medical unified multimodal model for
the simultaneous analysis of image understanding and generation tasks within a
single architecture. UniMedVL achieves superior performance on five medical
image understanding benchmarks, while matching specialized models in generation
quality across eight medical imaging modalities. Crucially, our unified
architecture enables bidirectional knowledge sharing: generation tasks enhance
visual understanding features, demonstrating that integrating traditionally
separate capabilities within a single medical framework unlocks improvements
across diverse medical vision-language tasks. Code is available at
https://github.com/uni-medical/UniMedVL.
A unified multimodal medical model integrates image understanding and generation, enhancing performance across various medical vision-language tasks.
UniMed-5M Progressive Curriculum Learning UniMedVL medical image understanding medical image generation multimodal pairs bidirectional knowledge sharing