Rare Disease Differential Diagnosis with Large Language Models at Scale:
  From Abdominal Actinomycosis to Wilson's Disease
Large language models (LLMs) have demonstrated impressive capabilities in
disease diagnosis. However, their effectiveness in identifying rarer diseases,
which are inherently more challenging to diagnose, remains an open question.
Rare disease performance is critical with the increasing use of LLMs in
healthcare settings. This is especially true if a primary care physician needs
to make a rarer prognosis from only a patient conversation so that they can
take the appropriate next step. To that end, several clinical decision support
systems are designed to support providers in rare disease identification. Yet
their utility is limited due to their lack of knowledge of common disorders and
difficulty of use.
  In this paper, we propose RareScale to combine the knowledge LLMs with expert
systems. We use jointly use an expert system and LLM to simulate rare disease
chats. This data is used to train a rare disease candidate predictor model.
Candidates from this smaller model are then used as additional inputs to
black-box LLM to make the final differential diagnosis. Thus, RareScale allows
for a balance between rare and common diagnoses. We present results on over 575
rare diseases, beginning with Abdominal Actinomycosis and ending with Wilson's
Disease. Our approach significantly improves the baseline performance of
black-box LLMs by over 17% in Top-5 accuracy. We also find that our candidate
generation performance is high (e.g. 88.8% on gpt-4o generated chats).
67bcdc646840d9686fcea432
Elliot Schumacher
extracted_confirmed
2025-02-24T20:54:12.397Z
64f74beb4db24c1ca9379afc
/avatars/c29b7d000ff65c590a5aec2d3262edd9.svg
Elliot Schumacher
user
elliotschu
67bcdc646840d9686fcea433
Dhruv Naik
67bcdc646840d9686fcea434
Anitha Kannan
Rare Disease Differential Diagnosis with Large Language Models at Scale:
  From Abdominal Actinomycosis to Wilson's Disease
Large language models (LLMs) have demonstrated impressive capabilities in
disease diagnosis. However, their effectiveness in identifying rarer diseases,
which are inherently more challenging to diagnose, remains an open question.
Rare disease performance is critical with the increasing use of LLMs in
healthcare settings. This is especially true if a primary care physician needs
to make a rarer prognosis from only a patient conversation so that they can
take the appropriate next step. To that end, several clinical decision support
systems are designed to support providers in rare disease identification. Yet
their utility is limited due to their lack of knowledge of common disorders and
difficulty of use.
  In this paper, we propose RareScale to combine the knowledge LLMs with expert
systems. We use jointly use an expert system and LLM to simulate rare disease
chats. This data is used to train a rare disease candidate predictor model.
Candidates from this smaller model are then used as additional inputs to
black-box LLM to make the final differential diagnosis. Thus, RareScale allows
for a balance between rare and common diagnoses. We present results on over 575
rare diseases, beginning with Abdominal Actinomycosis and ending with Wilson's
Disease. Our approach significantly improves the baseline performance of
black-box LLMs by over 17% in Top-5 accuracy. We also find that our candidate
generation performance is high (e.g. 88.8% on gpt-4o generated chats).
RareScale improves rare disease diagnosis performance by integrating expert systems with LLMs to balance rare and common diagnoses, enhancing Top-5 accuracy.
LLMs rare disease candidate predictor rare disease diagnosis disease simulation expert systems differential diagnosis