ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical
  Prediction?
Large Language Models (LLMs) hold great promise to revolutionize current
clinical systems for their superior capacities on medical text processing tasks
and medical licensing exams. Meanwhile, traditional ML models such as SVM and
XGBoost have still been mainly adopted in clinical prediction tasks. An
emerging question is Can LLMs beat traditional ML models in clinical
prediction? Thus, we build a new benchmark ClinicalBench to comprehensively
study the clinical predictive modeling capacities of both general-purpose and
medical LLMs, and compare them with traditional ML models. ClinicalBench
embraces three common clinical prediction tasks, two databases, 14
general-purpose LLMs, 8 medical LLMs, and 11 traditional ML models. Through
extensive empirical investigation, we discover that both general-purpose and
medical LLMs, even with different model scales, diverse prompting or
fine-tuning strategies, still cannot beat traditional ML models in clinical
prediction yet, shedding light on their potential deficiency in clinical
reasoning and decision-making. We call for caution when practitioners adopt
LLMs in clinical applications. ClinicalBench can be utilized to bridge the gap
between LLMs' development for healthcare and real-world clinical practice.
6736bdee81fd1225f7c1dc6d
Canyu Chen
claimed_verified
2024-11-15T09:21:19.591Z
6483af58571c2dcfa98cae82
https://cdn-avatars.huggingface.co/v1/production/uploads/6483af58571c2dcfa98cae82/d8Vnh_EG1KxeEoNAAQXHZ.jpeg
Canyu Chen
user
canyuchen
6736bdee81fd1225f7c1dc6e
Jian Yu
6736bdee81fd1225f7c1dc6f
Shan Chen
claimed_verified
2024-11-15T17:25:44.224Z
63600b93d9e4214b9a67807c
https://cdn-avatars.huggingface.co/v1/production/uploads/63600b93d9e4214b9a67807c/x_VE45ZPoaUFiryYl_5J5.png
Shan Chen
user
shanchen
6736bdee81fd1225f7c1dc70
Che Liu
claimed_verified
2025-02-28T12:17:57.848Z
631b9ff5824f2502e3557c7e
/avatars/076043c9dba07644a570692563ef8114.svg
liu
user
che111
6736bdee81fd1225f7c1dc71
Zhongwei Wan
admin_assigned
2024-11-15T14:04:59.950Z
64b20d0a372d4340773ff67e
/avatars/0f194f0f3345e8eb3f14fd99a1fb49f8.svg
Zhongwei Wang
user
hell-driver
6736bdee81fd1225f7c1dc72
Danielle Bitterman
admin_assigned
2024-11-15T14:01:30.067Z
6570cd33f3853f99bc305e55
https://cdn-avatars.huggingface.co/v1/production/uploads/6570cd33f3853f99bc305e55/70ZnaOtdtmy8DeWVJpCKb.png
Danielle Bitterman
user
daniellebitt
6736bdee81fd1225f7c1dc73
Fei Wang
6736bdee81fd1225f7c1dc74
Kai Shu
ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical
  Prediction?
Large Language Models (LLMs) hold great promise to revolutionize current
clinical systems for their superior capacities on medical text processing tasks
and medical licensing exams. Meanwhile, traditional ML models such as SVM and
XGBoost have still been mainly adopted in clinical prediction tasks. An
emerging question is Can LLMs beat traditional ML models in clinical
prediction? Thus, we build a new benchmark ClinicalBench to comprehensively
study the clinical predictive modeling capacities of both general-purpose and
medical LLMs, and compare them with traditional ML models. ClinicalBench
embraces three common clinical prediction tasks, two databases, 14
general-purpose LLMs, 8 medical LLMs, and 11 traditional ML models. Through
extensive empirical investigation, we discover that both general-purpose and
medical LLMs, even with different model scales, diverse prompting or
fine-tuning strategies, still cannot beat traditional ML models in clinical
prediction yet, shedding light on their potential deficiency in clinical
reasoning and decision-making. We call for caution when practitioners adopt
LLMs in clinical applications. ClinicalBench can be utilized to bridge the gap
between LLMs' development for healthcare and real-world clinical practice.
Despite their capabilities in text processing, existing large language models do not currently outperform traditional machine learning models in clinical prediction tasks.
Large Language Models LLMs ClinicalBench clinical prediction tasks medical text processing SVM XGBoost general-purpose LLMs medical LLMs prompting fine-tuning