Lingshu: A Generalist Foundation Model for Unified Multimodal Medical
  Understanding and Reasoning
Multimodal Large Language Models (MLLMs) have demonstrated impressive
capabilities in understanding common visual elements, largely due to their
large-scale datasets and advanced training strategies. However, their
effectiveness in medical applications remains limited due to the inherent
discrepancies between data and tasks in medical scenarios and those in the
general domain. Concretely, existing medical MLLMs face the following critical
limitations: (1) limited coverage of medical knowledge beyond imaging, (2)
heightened susceptibility to hallucinations due to suboptimal data curation
processes, (3) lack of reasoning capabilities tailored for complex medical
scenarios. To address these challenges, we first propose a comprehensive data
curation procedure that (1) efficiently acquires rich medical knowledge data
not only from medical imaging but also from extensive medical texts and
general-domain data; and (2) synthesizes accurate medical captions, visual
question answering (VQA), and reasoning samples. As a result, we build a
multimodal dataset enriched with extensive medical knowledge. Building on the
curated data, we introduce our medical-specialized MLLM: Lingshu. Lingshu
undergoes multi-stage training to embed medical expertise and enhance its
task-solving capabilities progressively. Besides, we preliminarily explore the
potential of applying reinforcement learning with verifiable rewards paradigm
to enhance Lingshu's medical reasoning ability. Additionally, we develop
MedEvalKit, a unified evaluation framework that consolidates leading multimodal
and textual medical benchmarks for standardized, fair, and efficient model
assessment. We evaluate the performance of Lingshu on three fundamental medical
tasks, multimodal QA, text-based QA, and medical report generation. The results
show that Lingshu consistently outperforms the existing open-source multimodal
models on most tasks ...
684795093ec10bdd8ab4de43
LASA Team
684795093ec10bdd8ab4de44
Weiwen Xu
claimed_verified
2025-06-10T08:44:07.459Z
64118689756b9e455c7eac62
/avatars/cdb3da22593facf545a0bafbf548b07e.svg
Xu Weiwen
user
xww033
684795093ec10bdd8ab4de45
Hou Pong Chan
claimed_verified
2025-06-10T08:44:05.163Z
604f67ef0fe8ff3ec13d71ef
https://cdn-avatars.huggingface.co/v1/production/uploads/604f67ef0fe8ff3ec13d71ef/KhUwWvZ3OJ9nEee3B-SXO.png
Hou Pong (Ken) Chan
user
kenchan0226
684795093ec10bdd8ab4de46
Long Li
claimed_verified
2025-07-15T19:12:01.942Z
6365d83ce7a78348d82572b0
/avatars/d50587902cced2c3640fd5de82ff78dd.svg
ll
user
jianghuyihei
684795093ec10bdd8ab4de47
Mahani Aljunied
684795093ec10bdd8ab4de48
Ruifeng Yuan
684795093ec10bdd8ab4de49
Jianyu Wang
claimed_verified
2025-06-10T08:44:03.340Z
61e09ec13a1781f66b4e9ae2
https://cdn-avatars.huggingface.co/v1/production/uploads/1642110635503-noauth.jpeg
Jianyu Wang
user
Jianyu
684795093ec10bdd8ab4de4a
Chenghao Xiao
claimed_verified
2025-06-10T10:59:59.163Z
63108cc834c7d77420b0fd68
/avatars/2721e573a417a8ec0b81ee048c4b42ba.svg
chenghao xiao
user
gowitheflow
684795093ec10bdd8ab4de4b
Guizhen Chen
684795093ec10bdd8ab4de4c
Chaoqun Liu
684795093ec10bdd8ab4de4d
Zhaodonghui Li
684795093ec10bdd8ab4de4e
Yu Sun
claimed_verified
2025-06-11T08:36:05.334Z
6723079ad1306fe9c76a1d29
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/b4BNPCeZs59MKxo1qmT6r.png
Yu Sun
user
YuSun-AI
684795093ec10bdd8ab4de4f
Junao Shen
684795093ec10bdd8ab4de50
Chaojun Wang
684795093ec10bdd8ab4de51
Jie Tan
684795093ec10bdd8ab4de52
Deli Zhao
684795093ec10bdd8ab4de53
Tingyang Xu
684795093ec10bdd8ab4de54
Hao Zhang
claimed_verified
2025-06-10T11:20:59.561Z
64b7cd74ff6d81ae297feded
/avatars/880fbc96cc093f5e901ce84f32a1d21d.svg
ZHANG HAO
user
26hzhang
684795093ec10bdd8ab4de55
Yu Rong
claimed_verified
2025-06-10T08:44:01.224Z
642eecbf9b2484d7d8526781
https://cdn-avatars.huggingface.co/v1/production/uploads/642eecbf9b2484d7d8526781/4IvGbd66s49Wx5pZyZGHA.png
Yu Rong
user
Swrooy
Lingshu: A Generalist Foundation Model for Unified Multimodal Medical
  Understanding and Reasoning
Multimodal Large Language Models (MLLMs) have demonstrated impressive
capabilities in understanding common visual elements, largely due to their
large-scale datasets and advanced training strategies. However, their
effectiveness in medical applications remains limited due to the inherent
discrepancies between data and tasks in medical scenarios and those in the
general domain. Concretely, existing medical MLLMs face the following critical
limitations: (1) limited coverage of medical knowledge beyond imaging, (2)
heightened susceptibility to hallucinations due to suboptimal data curation
processes, (3) lack of reasoning capabilities tailored for complex medical
scenarios. To address these challenges, we first propose a comprehensive data
curation procedure that (1) efficiently acquires rich medical knowledge data
not only from medical imaging but also from extensive medical texts and
general-domain data; and (2) synthesizes accurate medical captions, visual
question answering (VQA), and reasoning samples. As a result, we build a
multimodal dataset enriched with extensive medical knowledge. Building on the
curated data, we introduce our medical-specialized MLLM: Lingshu. Lingshu
undergoes multi-stage training to embed medical expertise and enhance its
task-solving capabilities progressively. Besides, we preliminarily explore the
potential of applying reinforcement learning with verifiable rewards paradigm
to enhance Lingshu's medical reasoning ability. Additionally, we develop
MedEvalKit, a unified evaluation framework that consolidates leading multimodal
and textual medical benchmarks for standardized, fair, and efficient model
assessment. We evaluate the performance of Lingshu on three fundamental medical
tasks, multimodal QA, text-based QA, and medical report generation. The results
show that Lingshu consistently outperforms the existing open-source multimodal
models on most tasks ...
A medical-specialized multimodal large language model, Lingshu, is introduced with enhanced data curation and reinforcement learning to address limitations in medical applications.
Multimodal Large Language Models MLLMs medical knowledge hallucinations data curation medical texts general-domain data accurate medical captions visual question answering VQA reasoning capabilities multi-stage training medical expertise reinforcement learning verifiable rewards paradigm MedEvalKit multimodal QA text-based QA medical report generation