CheXagent: Towards a Foundation Model for Chest X-Ray Interpretation
Chest X-rays (CXRs) are the most frequently performed imaging test in
clinical practice. Recent advances in the development of vision-language
foundation models (FMs) give rise to the possibility of performing automated
CXR interpretation, which can assist physicians with clinical decision-making
and improve patient outcomes. However, developing FMs that can accurately
interpret CXRs is challenging due to the (1) limited availability of
large-scale vision-language datasets in the medical image domain, (2) lack of
vision and language encoders that can capture the complexities of medical data,
and (3) absence of evaluation frameworks for benchmarking the abilities of FMs
on CXR interpretation. In this work, we address these challenges by first
introducing CheXinstruct - a large-scale instruction-tuning dataset
curated from 28 publicly-available datasets. We then present CheXagent -
an instruction-tuned FM capable of analyzing and summarizing CXRs. To build
CheXagent, we design a clinical large language model (LLM) for parsing
radiology reports, a vision encoder for representing CXR images, and a network
to bridge the vision and language modalities. Finally, we introduce
CheXbench - a novel benchmark designed to systematically evaluate FMs
across 8 clinically-relevant CXR interpretation tasks. Extensive quantitative
evaluations and qualitative reviews with five expert radiologists demonstrate
that CheXagent outperforms previously-developed general- and medical-domain FMs
on CheXbench tasks. Furthermore, in an effort to improve model transparency, we
perform a fairness evaluation across factors of sex, race and age to highlight
potential performance disparities. Our project is at
https://stanford-aimi.github.io/chexagent.html.
65af4222755c534def07a08b
Zhihong Chen
admin_assigned
2024-01-23T09:15:17.882Z
62e68ecff18bd3c3b532ddfd
/avatars/60ddbde83ab4be0277272a94e2e72e3a.svg
Zhihong Chen
user
zhjohnchan
65af4222755c534def07a08c
Maya Varma
admin_assigned
2024-01-23T09:16:02.233Z
6180c6e5000231f499c547c7
/avatars/0255b430be744a8da236860e2a00307e.svg
Maya Varma
user
mvarma
65af4222755c534def07a08d
Jean-Benoit Delbrouck
claimed_verified
2024-07-18T09:09:47.549Z
62716952bcef985363db8485
https://cdn-avatars.huggingface.co/v1/production/uploads/62716952bcef985363db8485/zJPPo5xlwZRJdEuwYsYKp.jpeg
JB D.
user
IAMJB
65af4222755c534def07a08e
Magdalini Paschali
claimed_verified
2024-01-24T08:09:11.441Z
65b00ea3399c0430e82ae284
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4Iovd06vaVhsZytvG066d.jpeg
Magda Paschali
user
magda-paschali
65af4222755c534def07a08f
Louis Blankemeier
admin_assigned
2024-01-23T09:16:22.179Z
6377044ea3b787faca443cc0
/avatars/5cd6cc70ad597dd83776018a452947d0.svg
Louis Blankemeier
user
louisblankemeier
65af4222755c534def07a090
Dave Van Veen
admin_assigned
2024-01-23T09:16:29.120Z
64e36dd9618cd90997e048f2
/avatars/2a239bed193982596114d52afea1bb97.svg
Dave Van Veen
user
davevanveen
65af4222755c534def07a091
Jeya Maria Jose Valanarasu
claimed_verified
2024-01-24T08:09:14.459Z
654eb5cbc67f60a3685e49e6
/avatars/047b2bfc768b1309b8e0363e5bcfbfbb.svg
Jeya Maria Jose Valanarasu
user
jmjose
65af4222755c534def07a092
Alaa Youssef
admin_assigned
2024-01-23T09:16:49.347Z
6526f2d0ad78d4a07a84cf24
/avatars/7e477a087098477b0d10517eab7f5a4e.svg
Alaa Youssef
user
Alaajoer
65af4222755c534def07a093
Joseph Paul Cohen
admin_assigned
2024-01-23T09:16:57.580Z
63a9d04f3453852ef53e1830
https://cdn-avatars.huggingface.co/v1/production/uploads/1672073530253-63a9d04f3453852ef53e1830.png
Joseph Paul Cohen
user
ieee8023
65af4222755c534def07a094
Eduardo Pontes Reis
admin_assigned
2024-01-23T09:17:04.835Z
60bc341d85b60a3459be9cac
/avatars/e0d3d2907504176451a0d22885eef997.svg
Eduardo Pontes Reis
user
edureisMD
65af4222755c534def07a095
Emily B. Tsai
65af4222755c534def07a096
Andrew Johnston
65af4222755c534def07a097
Cameron Olsen
65af4222755c534def07a098
Tanishq Mathew Abraham
admin_assigned
2024-01-23T09:26:44.198Z
6057b823861b9d53d9c4b8df
https://cdn-avatars.huggingface.co/v1/production/uploads/1625184855691-6057b823861b9d53d9c4b8df.jpeg
Tanishq Abraham
user
tmabraham
65af4222755c534def07a099
Sergios Gatidis
65af4222755c534def07a09a
Akshay S. Chaudhari
claimed_verified
2024-08-30T07:18:37.044Z
6236533b76c8a780323af640
/avatars/18078ad26aa44312a4927160216e5943.svg
Akshay Chaudhari
user
akshaysc
65af4222755c534def07a09b
Curtis Langlotz
claimed_verified
2024-08-30T07:18:38.658Z
62867edd1d99648808888d7b
/avatars/22d764782c5169634deec6f43ece7500.svg
Curt Langlotz
user
cplanglotz
CheXagent: Towards a Foundation Model for Chest X-Ray Interpretation
Chest X-rays (CXRs) are the most frequently performed imaging test in
clinical practice. Recent advances in the development of vision-language
foundation models (FMs) give rise to the possibility of performing automated
CXR interpretation, which can assist physicians with clinical decision-making
and improve patient outcomes. However, developing FMs that can accurately
interpret CXRs is challenging due to the (1) limited availability of
large-scale vision-language datasets in the medical image domain, (2) lack of
vision and language encoders that can capture the complexities of medical data,
and (3) absence of evaluation frameworks for benchmarking the abilities of FMs
on CXR interpretation. In this work, we address these challenges by first
introducing CheXinstruct - a large-scale instruction-tuning dataset
curated from 28 publicly-available datasets. We then present CheXagent -
an instruction-tuned FM capable of analyzing and summarizing CXRs. To build
CheXagent, we design a clinical large language model (LLM) for parsing
radiology reports, a vision encoder for representing CXR images, and a network
to bridge the vision and language modalities. Finally, we introduce
CheXbench - a novel benchmark designed to systematically evaluate FMs
across 8 clinically-relevant CXR interpretation tasks. Extensive quantitative
evaluations and qualitative reviews with five expert radiologists demonstrate
that CheXagent outperforms previously-developed general- and medical-domain FMs
on CheXbench tasks. Furthermore, in an effort to improve model transparency, we
perform a fairness evaluation across factors of sex, race and age to highlight
potential performance disparities. Our project is at
https://stanford-aimi.github.io/chexagent.html.
A large-scale instruction-tuning dataset and an instruction-tuned foundation model with a clinical large language model and vision encoder are introduced to automate CXR interpretation, outperforming existing models on clinical tasks and evaluated for fairness.
vision-language foundation models CheXinstruct CheXagent clinical large language model vision encoder CheXbench radiology reports CXR interpretation fairness evaluation