Exploring the Boundaries of GPT-4 in Radiology
The recent success of general-domain large language models (LLMs) has
significantly changed the natural language processing paradigm towards a
unified foundation model across domains and applications. In this paper, we
focus on assessing the performance of GPT-4, the most capable LLM so far, on
the text-based applications for radiology reports, comparing against
state-of-the-art (SOTA) radiology-specific models. Exploring various prompting
strategies, we evaluated GPT-4 on a diverse range of common radiology tasks and
we found GPT-4 either outperforms or is on par with current SOTA radiology
models. With zero-shot prompting, GPT-4 already obtains substantial gains
(approx 10% absolute improvement) over radiology models in temporal sentence
similarity classification (accuracy) and natural language inference (F_1).
For tasks that require learning dataset-specific style or schema (e.g. findings
summarisation), GPT-4 improves with example-based prompting and matches
supervised SOTA. Our extensive error analysis with a board-certified
radiologist shows GPT-4 has a sufficient level of radiology knowledge with only
occasional errors in complex context that require nuanced domain knowledge. For
findings summarisation, GPT-4 outputs are found to be overall comparable with
existing manually-written impressions.
65372de5e5ad8b46d53949b9
Qianchu Liu
admin_assigned
2023-10-24T08:41:09.123Z
617e7729129c9e67703ffe61
/avatars/f47ee9f2f0e2b1075bebf3682ee2f817.svg
qianchu liu
user
qianchu
65372de5e5ad8b46d53949ba
Stephanie Hyland
claimed_verified
2023-10-24T11:47:08.860Z
627554587aef4f8142f18bf9
https://cdn-avatars.huggingface.co/v1/production/uploads/627554587aef4f8142f18bf9/J8xOcG42lSCDDiewpzYob.jpeg
Stephanie
user
sthyland
65372de5e5ad8b46d53949bb
Shruthi Bannur
admin_assigned
2023-10-24T08:41:28.894Z
6274c513d400831ee0b8267c
/avatars/b4d183b7ab66cb7b16d6802c4850d8e8.svg
Shruthi Bannur
user
shruthib
65372de5e5ad8b46d53949bc
Kenza Bouzid
admin_assigned
2023-10-24T08:41:36.616Z
6511a4fbea5ccf1113bb545e
/avatars/8551b109249dd9c303ab17813297e09f.svg
Kenza Bouzid
user
kenza-bouzid
65372de5e5ad8b46d53949bd
Daniel C. Castro
65372de5e5ad8b46d53949be
Maria Teodora Wetscherek
65372de5e5ad8b46d53949bf
Robert Tinn
admin_assigned
2023-10-24T08:42:20.734Z
5f807ea93e94f16a85448947
/avatars/57f0d40178614e690864914015070fe0.svg
robert tinn
user
roberttinn
65372de5e5ad8b46d53949c0
Harshita Sharma
65372de5e5ad8b46d53949c1
Fernando Pérez-García
admin_assigned
2023-10-24T08:42:55.329Z
6278fbec17c069c72a11f35d
https://cdn-avatars.huggingface.co/v1/production/uploads/1669369101851-6278fbec17c069c72a11f35d.jpeg
Fernando Pérez-García
user
fepegar
65372de5e5ad8b46d53949c2
Anton Schwaighofer
65372de5e5ad8b46d53949c3
Pranav Rajpurkar
65372de5e5ad8b46d53949c4
Sameer Tajdin Khanna
65372de5e5ad8b46d53949c5
Hoifung Poon
65372de5e5ad8b46d53949c6
Naoto Usuyama
claimed_verified
2023-10-24T07:29:19.653Z
5f243561e923d665e616263f
https://cdn-avatars.huggingface.co/v1/production/uploads/1596208460315-noauth.jpeg
Naoto Usuyama
user
naotous
65372de5e5ad8b46d53949c7
Anja Thieme
65372de5e5ad8b46d53949c8
Aditya V. Nori
65372de5e5ad8b46d53949c9
Matthew P. Lungren
65372de5e5ad8b46d53949ca
Ozan Oktay
admin_assigned
2023-10-24T08:43:43.615Z
6274dd3eb43ccfeef481ce29
/avatars/9f5d2a25a74af3cb0a227f946041206d.svg
Ozan Oktay
user
ozanoktay
65372de5e5ad8b46d53949cb
Javier Alvarez-Valle
extracted_pending
2023-10-24T02:37:25.545Z
6278d875bba7c62cbaaf7693
/avatars/8d93e7c33acdfac853317d8fb5b0d485.svg
Javier Alvarez Valle
user
jaalvare
Exploring the Boundaries of GPT-4 in Radiology
The recent success of general-domain large language models (LLMs) has
significantly changed the natural language processing paradigm towards a
unified foundation model across domains and applications. In this paper, we
focus on assessing the performance of GPT-4, the most capable LLM so far, on
the text-based applications for radiology reports, comparing against
state-of-the-art (SOTA) radiology-specific models. Exploring various prompting
strategies, we evaluated GPT-4 on a diverse range of common radiology tasks and
we found GPT-4 either outperforms or is on par with current SOTA radiology
models. With zero-shot prompting, GPT-4 already obtains substantial gains
(approx 10% absolute improvement) over radiology models in temporal sentence
similarity classification (accuracy) and natural language inference (F_1).
For tasks that require learning dataset-specific style or schema (e.g. findings
summarisation), GPT-4 improves with example-based prompting and matches
supervised SOTA. Our extensive error analysis with a board-certified
radiologist shows GPT-4 has a sufficient level of radiology knowledge with only
occasional errors in complex context that require nuanced domain knowledge. For
findings summarisation, GPT-4 outputs are found to be overall comparable with
existing manually-written impressions.
GPT-4 performs effectively on various radiology tasks, matching or outperforming state-of-the-art radiology-specific models with prompting strategies.
GPT-4 large language models zero-shot prompting example-based prompting temporal sentence similarity classification natural language inference findings summarisation radiology-specific models error analysis board-certified radiologist