MedFuzz: Exploring the Robustness of Large Language Models in Medical
  Question Answering
Large language models (LLM) have achieved impressive performance on medical
question-answering benchmarks. However, high benchmark accuracy does not imply
that the performance generalizes to real-world clinical settings. Medical
question-answering benchmarks rely on assumptions consistent with quantifying
LLM performance but that may not hold in the open world of the clinic. Yet LLMs
learn broad knowledge that can help the LLM generalize to practical conditions
regardless of unrealistic assumptions in celebrated benchmarks. We seek to
quantify how well LLM medical question-answering benchmark performance
generalizes when benchmark assumptions are violated. Specifically, we present
an adversarial method that we call MedFuzz (for medical fuzzing). MedFuzz
attempts to modify benchmark questions in ways aimed at confounding the LLM. We
demonstrate the approach by targeting strong assumptions about patient
characteristics presented in the MedQA benchmark. Successful "attacks" modify a
benchmark item in ways that would be unlikely to fool a medical expert but
nonetheless "trick" the LLM into changing from a correct to an incorrect
answer. Further, we present a permutation test technique that can ensure a
successful attack is statistically significant. We show how to use performance
on a "MedFuzzed" benchmark, as well as individual successful attacks. The
methods show promise at providing insights into the ability of an LLM to
operate robustly in more realistic settings.
666919ea2de144452f920f45
Robert Osazuwa Ness
admin_assigned
2024-06-12T08:47:27.299Z
6384e1e9b2906edaf835dea4
/avatars/1087a75f7dbfd65b4d18d369d38eb5be.svg
Robert Osazuwa Ness
user
osazuwa
666919ea2de144452f920f46
Katie Matton
admin_assigned
2024-06-12T08:47:33.622Z
65c04c08262a04f94c4852cd
/avatars/17f727f8e7e7322a2af571d664ddff65.svg
Katie Matton
user
kmatton
666919ea2de144452f920f47
Hayden Helm
admin_assigned
2024-06-12T08:47:39.133Z
651726151ed9cd2cad93106f
/avatars/79db890fc8615700d9df05b264f3ca86.svg
Hayden Helm
user
hshelm
666919ea2de144452f920f48
Sheng Zhang
claimed_verified
2024-06-17T07:33:59.861Z
6234c11b7d5de9839bc44163
/avatars/3ca569596f9c7134e8d4b560a06ee1e7.svg
Sheng Zhang
user
shengz
666919ea2de144452f920f49
Junaid Bajwa
666919ea2de144452f920f4a
Carey E. Priebe
666919ea2de144452f920f4b
Eric Horvitz
admin_assigned
2024-06-12T08:48:21.339Z
6316576a76c93113fb15ae7f
/avatars/377045263dca803819289394da719ac9.svg
Eric Horvitz
user
skaviouz
MedFuzz: Exploring the Robustness of Large Language Models in Medical
  Question Answering
Large language models (LLM) have achieved impressive performance on medical
question-answering benchmarks. However, high benchmark accuracy does not imply
that the performance generalizes to real-world clinical settings. Medical
question-answering benchmarks rely on assumptions consistent with quantifying
LLM performance but that may not hold in the open world of the clinic. Yet LLMs
learn broad knowledge that can help the LLM generalize to practical conditions
regardless of unrealistic assumptions in celebrated benchmarks. We seek to
quantify how well LLM medical question-answering benchmark performance
generalizes when benchmark assumptions are violated. Specifically, we present
an adversarial method that we call MedFuzz (for medical fuzzing). MedFuzz
attempts to modify benchmark questions in ways aimed at confounding the LLM. We
demonstrate the approach by targeting strong assumptions about patient
characteristics presented in the MedQA benchmark. Successful "attacks" modify a
benchmark item in ways that would be unlikely to fool a medical expert but
nonetheless "trick" the LLM into changing from a correct to an incorrect
answer. Further, we present a permutation test technique that can ensure a
successful attack is statistically significant. We show how to use performance
on a "MedFuzzed" benchmark, as well as individual successful attacks. The
methods show promise at providing insights into the ability of an LLM to
operate robustly in more realistic settings.
MedFuzz, an adversarial method, demonstrates how well LLMs generalize to real-world clinical settings by modifying benchmark questions to confuse LLMs without fooling experts.
Large language models medical question-answering MedQA benchmark MedFuzz adversarial method permutation test robustness