MedCaseReasoning: Evaluating and learning diagnostic reasoning from
  clinical case reports
Doctors and patients alike increasingly use Large Language Models (LLMs) to
diagnose clinical cases. However, unlike domains such as math or coding, where
correctness can be objectively defined by the final answer, medical diagnosis
requires both the outcome and the reasoning process to be accurate. Currently,
widely used medical benchmarks like MedQA and MMLU assess only accuracy in the
final answer, overlooking the quality and faithfulness of the clinical
reasoning process. To address this limitation, we introduce MedCaseReasoning,
the first open-access dataset for evaluating LLMs on their ability to align
with clinician-authored diagnostic reasoning. The dataset includes 14,489
diagnostic question-and-answer cases, each paired with detailed reasoning
statements derived from open-access medical case reports. We evaluate
state-of-the-art reasoning LLMs on MedCaseReasoning and find significant
shortcomings in their diagnoses and reasoning: for instance, the top-performing
open-source model, DeepSeek-R1, achieves only 48% 10-shot diagnostic accuracy
and mentions only 64% of the clinician reasoning statements (recall). However,
we demonstrate that fine-tuning LLMs on the reasoning traces derived from
MedCaseReasoning significantly improves diagnostic accuracy and clinical
reasoning recall by an average relative gain of 29% and 41%, respectively. The
open-source dataset, code, and models are available at
https://github.com/kevinwu23/Stanford-MedCaseReasoning.
682ce238d3fa4ecfd67b99d0
Kevin Wu
682ce238d3fa4ecfd67b99d1
Eric Wu
claimed_verified
2025-05-21T08:41:38.271Z
61c4cf120cd370e91fc6f833
https://cdn-avatars.huggingface.co/v1/production/uploads/61c4cf120cd370e91fc6f833/Q7tyj2Q0qK5-FmEDZ34WI.jpeg
Eric Wu
user
ericwu09
682ce238d3fa4ecfd67b99d2
Rahul Thapa
682ce238d3fa4ecfd67b99d3
Kevin Wei
682ce238d3fa4ecfd67b99d4
Angela Zhang
682ce238d3fa4ecfd67b99d5
Arvind Suresh
682ce238d3fa4ecfd67b99d6
Jacqueline J. Tao
682ce238d3fa4ecfd67b99d7
Min Woo Sun
claimed_verified
2025-05-20T20:41:36.732Z
65ac61120844d9e0d67a9f89
https://cdn-avatars.huggingface.co/v1/production/uploads/65ac61120844d9e0d67a9f89/OzA_yxFoeuyhmc3xdYYdP.jpeg
Min Woo Sun
user
minwoosun
682ce238d3fa4ecfd67b99d8
Alejandro Lozano
682ce238d3fa4ecfd67b99d9
James Zou
MedCaseReasoning: Evaluating and learning diagnostic reasoning from
  clinical case reports
Doctors and patients alike increasingly use Large Language Models (LLMs) to
diagnose clinical cases. However, unlike domains such as math or coding, where
correctness can be objectively defined by the final answer, medical diagnosis
requires both the outcome and the reasoning process to be accurate. Currently,
widely used medical benchmarks like MedQA and MMLU assess only accuracy in the
final answer, overlooking the quality and faithfulness of the clinical
reasoning process. To address this limitation, we introduce MedCaseReasoning,
the first open-access dataset for evaluating LLMs on their ability to align
with clinician-authored diagnostic reasoning. The dataset includes 14,489
diagnostic question-and-answer cases, each paired with detailed reasoning
statements derived from open-access medical case reports. We evaluate
state-of-the-art reasoning LLMs on MedCaseReasoning and find significant
shortcomings in their diagnoses and reasoning: for instance, the top-performing
open-source model, DeepSeek-R1, achieves only 48% 10-shot diagnostic accuracy
and mentions only 64% of the clinician reasoning statements (recall). However,
we demonstrate that fine-tuning LLMs on the reasoning traces derived from
MedCaseReasoning significantly improves diagnostic accuracy and clinical
reasoning recall by an average relative gain of 29% and 41%, respectively. The
open-source dataset, code, and models are available at
https://github.com/kevinwu23/Stanford-MedCaseReasoning.
MedCaseReasoning is an open-access dataset that evaluates LLMs on diagnostic accuracy and clinical reasoning, revealing that fine-tuning on reasoning traces improves model performance.
Large Language Models (LLMs) medical benchmarks MedQA MMLU MedCaseReasoning diagnostic question-and-answer cases reasoning LLMs DeepSeek-R1 diagnostic accuracy recall fine-tuning reasoning traces