MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations
  in Large Language Models
Advancements in Large Language Models (LLMs) and their increasing use in
medical question-answering necessitate rigorous evaluation of their
reliability. A critical challenge lies in hallucination, where models generate
plausible yet factually incorrect outputs. In the medical domain, this poses
serious risks to patient safety and clinical decision-making. To address this,
we introduce MedHallu, the first benchmark specifically designed for medical
hallucination detection. MedHallu comprises 10,000 high-quality question-answer
pairs derived from PubMedQA, with hallucinated answers systematically generated
through a controlled pipeline. Our experiments show that state-of-the-art LLMs,
including GPT-4o, Llama-3.1, and the medically fine-tuned UltraMedical,
struggle with this binary hallucination detection task, with the best model
achieving an F1 score as low as 0.625 for detecting "hard" category
hallucinations. Using bidirectional entailment clustering, we show that
harder-to-detect hallucinations are semantically closer to ground truth.
Through experiments, we also show incorporating domain-specific knowledge and
introducing a "not sure" category as one of the answer categories improves the
precision and F1 scores by up to 38% relative to baselines.
67b7e1a0e5dcadcedba108e8
Shrey Pandit
extracted_confirmed
2025-02-22T19:42:39.549Z
648749094dea003c6dae810f
https://cdn-avatars.huggingface.co/v1/production/uploads/648749094dea003c6dae810f/gHUHSBt1zrt8wjO1YwTNu.jpeg
Shrey Pandit
user
SP2001
67b7e1a0e5dcadcedba108e9
Jiawei Xu
67b7e1a0e5dcadcedba108ea
Junyuan Hong
admin_assigned
2025-02-24T15:58:00.251Z
6400cf982b67d27affce2d89
https://cdn-avatars.huggingface.co/v1/production/uploads/6400cf982b67d27affce2d89/Vs042d2M-iV2wk9Q_Jqh3.jpeg
Junyuan Hong
user
jyhong836
67b7e1a0e5dcadcedba108eb
Zhangyang Wang
67b7e1a0e5dcadcedba108ec
Tianlong Chen
67b7e1a0e5dcadcedba108ed
Kaidi Xu
admin_assigned
2025-02-24T16:00:54.855Z
65aaa5b4d2adc31ee3eab350
https://cdn-avatars.huggingface.co/v1/production/uploads/65aaa5b4d2adc31ee3eab350/vzDrR7JVCEShn_Vk-NCgT.jpeg
Kaidi Xu
user
KaidiXu1
67b7e1a0e5dcadcedba108ee
Ying Ding
MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations
  in Large Language Models
Advancements in Large Language Models (LLMs) and their increasing use in
medical question-answering necessitate rigorous evaluation of their
reliability. A critical challenge lies in hallucination, where models generate
plausible yet factually incorrect outputs. In the medical domain, this poses
serious risks to patient safety and clinical decision-making. To address this,
we introduce MedHallu, the first benchmark specifically designed for medical
hallucination detection. MedHallu comprises 10,000 high-quality question-answer
pairs derived from PubMedQA, with hallucinated answers systematically generated
through a controlled pipeline. Our experiments show that state-of-the-art LLMs,
including GPT-4o, Llama-3.1, and the medically fine-tuned UltraMedical,
struggle with this binary hallucination detection task, with the best model
achieving an F1 score as low as 0.625 for detecting "hard" category
hallucinations. Using bidirectional entailment clustering, we show that
harder-to-detect hallucinations are semantically closer to ground truth.
Through experiments, we also show incorporating domain-specific knowledge and
introducing a "not sure" category as one of the answer categories improves the
precision and F1 scores by up to 38% relative to baselines.
A benchmark called MedHallu evaluates the reliability of large language models in medical question-answering by detecting hallucinations, with improvements shown via bidirectional entailment clustering and domain-specific knowledge.
Large Language Models (LLMs) medical question-answering hallucination MedHallu PubMedQA F1 score bidirectional entailment clustering domain-specific knowledge "not sure" category