Towards Expert-Level Medical Question Answering with Large Language
  Models
Recent artificial intelligence (AI) systems have reached milestones in "grand
challenges" ranging from Go to protein-folding. The capability to retrieve
medical knowledge, reason over it, and answer medical questions comparably to
physicians has long been viewed as one such grand challenge.
  Large language models (LLMs) have catalyzed significant progress in medical
question answering; Med-PaLM was the first model to exceed a "passing" score in
US Medical Licensing Examination (USMLE) style questions with a score of 67.2%
on the MedQA dataset. However, this and other prior work suggested significant
room for improvement, especially when models' answers were compared to
clinicians' answers. Here we present Med-PaLM 2, which bridges these gaps by
leveraging a combination of base LLM improvements (PaLM 2), medical domain
finetuning, and prompting strategies including a novel ensemble refinement
approach.
  Med-PaLM 2 scored up to 86.5% on the MedQA dataset, improving upon Med-PaLM
by over 19% and setting a new state-of-the-art. We also observed performance
approaching or exceeding state-of-the-art across MedMCQA, PubMedQA, and MMLU
clinical topics datasets.
  We performed detailed human evaluations on long-form questions along multiple
axes relevant to clinical applications. In pairwise comparative ranking of 1066
consumer medical questions, physicians preferred Med-PaLM 2 answers to those
produced by physicians on eight of nine axes pertaining to clinical utility (p
< 0.001). We also observed significant improvements compared to Med-PaLM on
every evaluation axis (p < 0.001) on newly introduced datasets of 240 long-form
"adversarial" questions to probe LLM limitations.
  While further studies are necessary to validate the efficacy of these models
in real-world settings, these results highlight rapid progress towards
physician-level performance in medical question answering.
64642c4c7572c66a8e649948
Karan Singhal
64642c4c7572c66a8e649949
Tao Tu
64642c4c7572c66a8e64994a
Juraj Gottweis
admin_assigned
2023-05-26T09:21:14.750Z
6381e100cae34727b9ca33ce
/avatars/f4e047e82e41ec507c6f51aef7560d87.svg
Juraj Gottweis
user
juro
64642c4c7572c66a8e64994b
Rory Sayres
64642c4c7572c66a8e64994c
Ellery Wulczyn
64642c4c7572c66a8e64994d
Le Hou
claimed_verified
2023-05-19T10:28:09.156Z
626b2402c804c432c1b5004c
/avatars/a119644422335e260af0ca4ef6399870.svg
Le Hou
user
solooobear
64642c4c7572c66a8e64994e
Kevin Clark
64642c4c7572c66a8e64994f
Stephen Pfohl
64642c4c7572c66a8e649950
Heather Cole-Lewis
64642c4c7572c66a8e649951
Darlene Neal
64642c4c7572c66a8e649952
Mike Schaekermann
64642c4c7572c66a8e649953
Amy Wang
64642c4c7572c66a8e649954
Mohamed Amin
64642c4c7572c66a8e649955
Sami Lachgar
64642c4c7572c66a8e649956
Philip Mansfield
64642c4c7572c66a8e649957
Sushant Prakash
64642c4c7572c66a8e649958
Bradley Green
64642c4c7572c66a8e649959
Ewa Dominowska
64642c4c7572c66a8e64995a
Blaise Aguera y Arcas
64642c4c7572c66a8e64995b
Nenad Tomasev
64642c4c7572c66a8e64995c
Yun Liu
64642c4c7572c66a8e64995d
Renee Wong
64642c4c7572c66a8e64995e
Christopher Semturs
64642c4c7572c66a8e64995f
S. Sara Mahdavi
64642c4c7572c66a8e649960
Joelle Barral
64642c4c7572c66a8e649961
Dale Webster
64642c4c7572c66a8e649962
Greg S. Corrado
64642c4c7572c66a8e649963
Yossi Matias
64642c4c7572c66a8e649964
Shekoofeh Azizi
64642c4c7572c66a8e649965
Alan Karthikesalingam
64642c4c7572c66a8e649966
Vivek Natarajan
admin_assigned
2023-05-26T09:24:24.776Z
617a6af2941993035fbaf29c
/avatars/a389671494782f06ff8502c934090cc0.svg
Vivek Natarajan
user
vivnat
Towards Expert-Level Medical Question Answering with Large Language
  Models
Recent artificial intelligence (AI) systems have reached milestones in "grand
challenges" ranging from Go to protein-folding. The capability to retrieve
medical knowledge, reason over it, and answer medical questions comparably to
physicians has long been viewed as one such grand challenge.
  Large language models (LLMs) have catalyzed significant progress in medical
question answering; Med-PaLM was the first model to exceed a "passing" score in
US Medical Licensing Examination (USMLE) style questions with a score of 67.2%
on the MedQA dataset. However, this and other prior work suggested significant
room for improvement, especially when models' answers were compared to
clinicians' answers. Here we present Med-PaLM 2, which bridges these gaps by
leveraging a combination of base LLM improvements (PaLM 2), medical domain
finetuning, and prompting strategies including a novel ensemble refinement
approach.
  Med-PaLM 2 scored up to 86.5% on the MedQA dataset, improving upon Med-PaLM
by over 19% and setting a new state-of-the-art. We also observed performance
approaching or exceeding state-of-the-art across MedMCQA, PubMedQA, and MMLU
clinical topics datasets.
  We performed detailed human evaluations on long-form questions along multiple
axes relevant to clinical applications. In pairwise comparative ranking of 1066
consumer medical questions, physicians preferred Med-PaLM 2 answers to those
produced by physicians on eight of nine axes pertaining to clinical utility (p
< 0.001). We also observed significant improvements compared to Med-PaLM on
every evaluation axis (p < 0.001) on newly introduced datasets of 240 long-form
"adversarial" questions to probe LLM limitations.
  While further studies are necessary to validate the efficacy of these models
in real-world settings, these results highlight rapid progress towards
physician-level performance in medical question answering.
Med-PaLM 2, built on PaLM 2 and enhanced with medical fine-tuning and ensemble refinement, significantly improves physician-level performance in medical question answering by scoring 86.5% on MedQA and excelling on other clinical datasets.
large language models LLMs Med-PaLM Med-PaLM 2 PaLM 2 medical domain finetuning prompting strategies ensemble refinement MedQA dataset MedMCQA PubMedQA MMLU clinical topics human evaluations clinical utility adversarial questions