Capabilities of Gemini Models in Medicine
Excellence in a wide variety of medical applications poses considerable
challenges for AI, requiring advanced reasoning, access to up-to-date medical
knowledge and understanding of complex multimodal data. Gemini models, with
strong general capabilities in multimodal and long-context reasoning, offer
exciting possibilities in medicine. Building on these core strengths of Gemini,
we introduce Med-Gemini, a family of highly capable multimodal models that are
specialized in medicine with the ability to seamlessly use web search, and that
can be efficiently tailored to novel modalities using custom encoders. We
evaluate Med-Gemini on 14 medical benchmarks, establishing new state-of-the-art
(SoTA) performance on 10 of them, and surpass the GPT-4 model family on every
benchmark where a direct comparison is viable, often by a wide margin. On the
popular MedQA (USMLE) benchmark, our best-performing Med-Gemini model achieves
SoTA performance of 91.1% accuracy, using a novel uncertainty-guided search
strategy. On 7 multimodal benchmarks including NEJM Image Challenges and MMMU
(health & medicine), Med-Gemini improves over GPT-4V by an average relative
margin of 44.5%. We demonstrate the effectiveness of Med-Gemini's long-context
capabilities through SoTA performance on a needle-in-a-haystack retrieval task
from long de-identified health records and medical video question answering,
surpassing prior bespoke methods using only in-context learning. Finally,
Med-Gemini's performance suggests real-world utility by surpassing human
experts on tasks such as medical text summarization, alongside demonstrations
of promising potential for multimodal medical dialogue, medical research and
education. Taken together, our results offer compelling evidence for
Med-Gemini's potential, although further rigorous evaluation will be crucial
before real-world deployment in this safety-critical domain.
663059ade649907fb14aebee
Khaled Saab
663059ade649907fb14aebef
Tao Tu
663059ade649907fb14aebf0
Wei-Hung Weng
663059ade649907fb14aebf1
Ryutaro Tanno
663059ade649907fb14aebf2
David Stutz
663059ade649907fb14aebf3
Ellery Wulczyn
663059ade649907fb14aebf4
Fan Zhang
663059ade649907fb14aebf5
Tim Strother
663059ade649907fb14aebf6
Chunjong Park
663059ade649907fb14aebf7
Elahe Vedadi
663059ade649907fb14aebf8
Juanma Zambrano Chaves
663059ade649907fb14aebf9
Szu-Yeu Hu
admin_assigned
2024-04-30T08:10:37.627Z
6626acc7b24310dd5feee335
/avatars/b2a950ae946887935233d54d9374118d.svg
Szu-Yeu Hu
user
sdcjimmy
663059ade649907fb14aebfa
Mike Schaekermann
admin_assigned
2024-04-30T08:10:43.861Z
64ebb75f171e6c9862d261c9
/avatars/b7750f5d6281f906cfa5c717a44e2e13.svg
Mike Schaekermann
user
mikeschaekermann
663059ade649907fb14aebfb
Aishwarya Kamath
admin_assigned
2024-04-30T08:10:52.656Z
636996b9d322a76e1ea7e59d
/avatars/100cee613df9914c414f51be0ca86810.svg
Aishwarya Kamath
user
ashkamath
663059ade649907fb14aebfc
Yong Cheng
663059ade649907fb14aebfd
David G. T. Barrett
663059ade649907fb14aebfe
Cathy Cheung
663059ade649907fb14aebff
Basil Mustafa
admin_assigned
2024-04-30T08:11:50.424Z
63cfcad6e23b90128c66685c
https://cdn-avatars.huggingface.co/v1/production/uploads/1674562206311-noauth.png
Basil Mustafa
user
BasilMustafa
663059ade649907fb14aec00
Anil Palepu
663059ade649907fb14aec01
Daniel McDuff
663059ade649907fb14aec02
Le Hou
admin_assigned
2024-04-30T08:12:39.746Z
626b2402c804c432c1b5004c
/avatars/a119644422335e260af0ca4ef6399870.svg
Le Hou
user
solooobear
663059ade649907fb14aec03
Tomer Golany
admin_assigned
2024-04-30T08:12:46.302Z
64f06d58a5aeea35fcbf4bae
/avatars/4966ab54ca7074c220b706a0d159c9b7.svg
Tomer Golany
user
tomergolany
663059ade649907fb14aec04
Luyang Liu
admin_assigned
2024-04-30T08:13:33.141Z
65d8fa4e16cebdd689fa5587
/avatars/9646f22ae16653af47ea38af0839ae7b.svg
Luyang Liu
user
luyangl
663059ade649907fb14aec05
Jean-baptiste Alayrac
admin_assigned
2024-04-30T08:13:44.292Z
6253ee39e4e98393660b5c35
/avatars/9c032f6a0729bfe5c16b3affe190834d.svg
Jean-Baptiste Alayrac
user
jalayrac
663059ade649907fb14aec06
Neil Houlsby
admin_assigned
2024-04-30T08:13:51.325Z
64a2c7f263f69fb98d3bfdb4
/avatars/a362a236c0654b7605dcb7673e309335.svg
Neil Houlsby
user
neilhoulsby
663059ade649907fb14aec07
Nenad Tomasev
663059ade649907fb14aec08
Jan Freyberg
admin_assigned
2024-04-30T08:14:02.247Z
629656557d586f66c39206e7
/avatars/91245f3b6a2eccebef872ed00bdfea73.svg
Jan Freyberg
user
jfreyberg
663059ade649907fb14aec09
Charles Lau
663059ade649907fb14aec0a
Jonas Kemp
663059ade649907fb14aec0b
Jeremy Lai
663059ade649907fb14aec0c
Shekoofeh Azizi
663059ade649907fb14aec0d
Kimberly Kanada
663059ade649907fb14aec0e
SiWai Man
663059ade649907fb14aec0f
Kavita Kulkarni
663059ade649907fb14aec10
Ruoxi Sun
admin_assigned
2024-04-30T08:15:21.516Z
653093ad471261a41241e048
/avatars/54263ce75b102b81211d27b696920870.svg
Ruoxi Sun
user
brycesun
663059ade649907fb14aec11
Siamak Shakeri
663059ade649907fb14aec12
Luheng He
663059ade649907fb14aec13
Ben Caine
663059ade649907fb14aec14
Albert Webson
admin_assigned
2024-04-30T08:15:46.876Z
617ada70ff3db6021d069798
/avatars/f14c59fd56a49dd9a925285a4fb66fff.svg
Albert Webson
user
awebson
663059ade649907fb14aec15
Natasha Latysheva
663059ade649907fb14aec16
Melvin Johnson
663059ade649907fb14aec17
Philip Mansfield
663059ade649907fb14aec18
Jian Lu
663059ade649907fb14aec19
Ehud Rivlin
663059ade649907fb14aec1a
Jesper Anderson
663059ade649907fb14aec1b
Bradley Green
admin_assigned
2024-04-30T08:16:36.781Z
63417e33dd8853dc7e33c6a1
/avatars/3a0bcd81d2dc1b032e445997485ebb49.svg
bradley green
user
cybercow2077
663059ade649907fb14aec1c
Renee Wong
admin_assigned
2024-04-30T08:16:45.053Z
6397fb67a51b5dbfc93ca0d1
/avatars/b254d4df8e4158e2dbd047ee077440e3.svg
Wong
user
reneewong60
663059ade649907fb14aec1d
Jonathan Krause
663059ade649907fb14aec1e
Jonathon Shlens
663059ade649907fb14aec1f
Ewa Dominowska
663059ade649907fb14aec20
S. M. Ali Eslami
663059ade649907fb14aec21
Claire Cui
663059ade649907fb14aec22
Oriol Vinyals
663059ade649907fb14aec23
Koray Kavukcuoglu
663059ade649907fb14aec24
James Manyika
663059ade649907fb14aec25
Jeff Dean
663059ade649907fb14aec26
Demis Hassabis
663059ade649907fb14aec27
Yossi Matias
663059ade649907fb14aec28
Dale Webster
admin_assigned
2024-04-30T08:18:04.482Z
63124715bbaa385279cebcec
/avatars/120869c6dd8ed8cfdd4facb688a8f0af.svg
Dale Webster
user
allany
663059ade649907fb14aec29
Joelle Barral
663059ade649907fb14aec2a
Greg Corrado
663059ade649907fb14aec2b
Christopher Semturs
663059ade649907fb14aec2c
S. Sara Mahdavi
663059ade649907fb14aec2d
Juraj Gottweis
admin_assigned
2024-04-30T08:17:30.136Z
6381e100cae34727b9ca33ce
/avatars/f4e047e82e41ec507c6f51aef7560d87.svg
Juraj Gottweis
user
juro
663059ade649907fb14aec2e
Alan Karthikesalingam
663059ade649907fb14aec2f
Vivek Natarajan
admin_assigned
2024-04-30T08:17:17.630Z
617a6af2941993035fbaf29c
/avatars/a389671494782f06ff8502c934090cc0.svg
Vivek Natarajan
user
vivnat
Capabilities of Gemini Models in Medicine
Excellence in a wide variety of medical applications poses considerable
challenges for AI, requiring advanced reasoning, access to up-to-date medical
knowledge and understanding of complex multimodal data. Gemini models, with
strong general capabilities in multimodal and long-context reasoning, offer
exciting possibilities in medicine. Building on these core strengths of Gemini,
we introduce Med-Gemini, a family of highly capable multimodal models that are
specialized in medicine with the ability to seamlessly use web search, and that
can be efficiently tailored to novel modalities using custom encoders. We
evaluate Med-Gemini on 14 medical benchmarks, establishing new state-of-the-art
(SoTA) performance on 10 of them, and surpass the GPT-4 model family on every
benchmark where a direct comparison is viable, often by a wide margin. On the
popular MedQA (USMLE) benchmark, our best-performing Med-Gemini model achieves
SoTA performance of 91.1% accuracy, using a novel uncertainty-guided search
strategy. On 7 multimodal benchmarks including NEJM Image Challenges and MMMU
(health & medicine), Med-Gemini improves over GPT-4V by an average relative
margin of 44.5%. We demonstrate the effectiveness of Med-Gemini's long-context
capabilities through SoTA performance on a needle-in-a-haystack retrieval task
from long de-identified health records and medical video question answering,
surpassing prior bespoke methods using only in-context learning. Finally,
Med-Gemini's performance suggests real-world utility by surpassing human
experts on tasks such as medical text summarization, alongside demonstrations
of promising potential for multimodal medical dialogue, medical research and
education. Taken together, our results offer compelling evidence for
Med-Gemini's potential, although further rigorous evaluation will be crucial
before real-world deployment in this safety-critical domain.
Med-Gemini, a family of specialized multimodal AI models, achieves state-of-the-art performance across various medical benchmarks, surpassing GPT-4 and demonstrating superior long-context and uncertainty-guided search capabilities.
multimodal diamond models long-context reasoning Med-Gemini web search custom encoders state-of-the-art SoTA MedQA uncertainty-guided search NEJM Image Challenges MMMU GPT-4 needle-in-a-haystack retrieval medical video question answering in-context learning medical text summarization multimodal medical dialogue medical research medical education