A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?
Large language models (LLMs) have exhibited remarkable capabilities across
various domains and tasks, pushing the boundaries of our knowledge in learning
and cognition. The latest model, OpenAI's o1, stands out as the first LLM with
an internalized chain-of-thought technique using reinforcement learning
strategies. While it has demonstrated surprisingly strong capabilities on
various general language tasks, its performance in specialized fields such as
medicine remains unknown. To this end, this report provides a comprehensive
exploration of o1 on different medical scenarios, examining 3 key aspects:
understanding, reasoning, and multilinguality. Specifically, our evaluation
encompasses 6 tasks using data from 37 medical datasets, including two newly
constructed and more challenging question-answering (QA) tasks based on
professional medical quizzes from the New England Journal of Medicine (NEJM)
and The Lancet. These datasets offer greater clinical relevance compared to
standard medical QA benchmarks such as MedQA, translating more effectively into
real-world clinical utility. Our analysis of o1 suggests that the enhanced
reasoning ability of LLMs may (significantly) benefit their capability to
understand various medical instructions and reason through complex clinical
scenarios. Notably, o1 surpasses the previous GPT-4 in accuracy by an average
of 6.2% and 6.6% across 19 datasets and two newly created complex QA scenarios.
But meanwhile, we identify several weaknesses in both the model capability and
the existing evaluation protocols, including hallucination, inconsistent
multilingual ability, and discrepant metrics for evaluation. We release our raw
data and model outputs at https://ucsc-vlaa.github.io/o1_medicine/ for future
research.
66f22c153a6d90d35ef1ac71
Yunfei Xie
claimed_verified
2024-10-11T07:44:30.869Z
65f6afa36ea40b9a29dd45ca
/avatars/a80a25fa70a89a3bf3d50ea2866f0df1.svg
Yunfei Xie
user
yunfeixie
66f22c153a6d90d35ef1ac72
Juncheng Wu
66f22c153a6d90d35ef1ac73
Haoqin Tu
claimed_verified
2024-12-02T08:51:57.148Z
604ae011caabafacfa48e3de
https://cdn-avatars.huggingface.co/v1/production/uploads/1615519738679-noauth.jpeg
Haoqin Tu
user
PahaII
66f22c153a6d90d35ef1ac74
Siwei Yang
claimed_verified
2025-04-22T10:17:31.884Z
6433fd1095b8ab049385d2b3
/avatars/718117cd4631b7c297add7e559032b2f.svg
Siwei Yang
user
LAOS-Y
66f22c153a6d90d35ef1ac75
Bingchen Zhao
claimed_verified
2024-09-24T07:52:44.028Z
62dcd71075e9787ec5aa41ba
/avatars/f37ce036b76180ed0fa004f9c8c09363.svg
Bingchen Zhao
user
tennant
66f22c153a6d90d35ef1ac76
Yongshuo Zong
claimed_verified
2024-09-24T07:52:42.234Z
643d5de2c17bfb2256f14b4b
/avatars/1124128ce69ae6868fd1a2c06420d72f.svg
Yongshuo Zong
user
ys-zong
66f22c153a6d90d35ef1ac77
Qiao Jin
claimed_verified
2025-02-07T16:17:04.847Z
633f51417f69125dcea06aca
/avatars/2b7fe0d262e336932741d0b498987ae5.svg
Qiao Jin
user
qiaojin
66f22c153a6d90d35ef1ac78
Cihang Xie
claimed_verified
2025-07-31T08:17:17.617Z
645eb61da3c5cd8a16efffff
/avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
Cihang Xie
user
cihangxie
66f22c153a6d90d35ef1ac79
Yuyin Zhou
A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?
Large language models (LLMs) have exhibited remarkable capabilities across
various domains and tasks, pushing the boundaries of our knowledge in learning
and cognition. The latest model, OpenAI's o1, stands out as the first LLM with
an internalized chain-of-thought technique using reinforcement learning
strategies. While it has demonstrated surprisingly strong capabilities on
various general language tasks, its performance in specialized fields such as
medicine remains unknown. To this end, this report provides a comprehensive
exploration of o1 on different medical scenarios, examining 3 key aspects:
understanding, reasoning, and multilinguality. Specifically, our evaluation
encompasses 6 tasks using data from 37 medical datasets, including two newly
constructed and more challenging question-answering (QA) tasks based on
professional medical quizzes from the New England Journal of Medicine (NEJM)
and The Lancet. These datasets offer greater clinical relevance compared to
standard medical QA benchmarks such as MedQA, translating more effectively into
real-world clinical utility. Our analysis of o1 suggests that the enhanced
reasoning ability of LLMs may (significantly) benefit their capability to
understand various medical instructions and reason through complex clinical
scenarios. Notably, o1 surpasses the previous GPT-4 in accuracy by an average
of 6.2% and 6.6% across 19 datasets and two newly created complex QA scenarios.
But meanwhile, we identify several weaknesses in both the model capability and
the existing evaluation protocols, including hallucination, inconsistent
multilingual ability, and discrepant metrics for evaluation. We release our raw
data and model outputs at https://ucsc-vlaa.github.io/o1_medicine/ for future
research.
OpenAI's o1 demonstrates strong performance in medical tasks but shows weaknesses in reasoning, multilinguality, and evaluation metrics compared to GPT-4.
chain-of-thought technique reinforcement learning question-answering medical datasets New England Journal of Medicine The Lancet MedQA hallucination inconsistent multilingual ability discrepant metrics