Comprehensive and Practical Evaluation of Retrieval-Augmented Generation
  Systems for Medical Question Answering
Retrieval-augmented generation (RAG) has emerged as a promising approach to
enhance the performance of large language models (LLMs) in knowledge-intensive
tasks such as those from medical domain. However, the sensitive nature of the
medical domain necessitates a completely accurate and trustworthy system. While
existing RAG benchmarks primarily focus on the standard retrieve-answer
setting, they overlook many practical scenarios that measure crucial aspects of
a reliable medical system. This paper addresses this gap by providing a
comprehensive evaluation framework for medical question-answering (QA) systems
in a RAG setting for these situations, including sufficiency, integration, and
robustness. We introduce Medical Retrieval-Augmented Generation Benchmark
(MedRGB) that provides various supplementary elements to four medical QA
datasets for testing LLMs' ability to handle these specific scenarios.
Utilizing MedRGB, we conduct extensive evaluations of both state-of-the-art
commercial LLMs and open-source models across multiple retrieval conditions.
Our experimental results reveals current models' limited ability to handle
noise and misinformation in the retrieved documents. We further analyze the
LLMs' reasoning processes to provides valuable insights and future directions
for developing RAG systems in this critical medical domain.
673c338be17a2c7a97e8b8c3
Nghia Trung Ngo
admin_assigned
2024-11-19T14:21:03.206Z
658a8208c4b2004663d82daf
/avatars/af6bcee06aec82602c2b931f79c008e7.svg
Nghia Trung Ngo
user
ntnghia1811
673c338be17a2c7a97e8b8c4
Chien Van Nguyen
673c338be17a2c7a97e8b8c5
Franck Dernoncourt
extracted_confirmed
2024-11-19T06:43:48.817Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
673c338be17a2c7a97e8b8c6
Thien Huu Nguyen
admin_assigned
2024-11-19T14:21:15.211Z
64804fad8c6a3b8f11f73912
/avatars/61e37a91d4bba35fda9bf52aadd87745.svg
Thien Huu Nguyen
user
anoperson
Comprehensive and Practical Evaluation of Retrieval-Augmented Generation
  Systems for Medical Question Answering
Retrieval-augmented generation (RAG) has emerged as a promising approach to
enhance the performance of large language models (LLMs) in knowledge-intensive
tasks such as those from medical domain. However, the sensitive nature of the
medical domain necessitates a completely accurate and trustworthy system. While
existing RAG benchmarks primarily focus on the standard retrieve-answer
setting, they overlook many practical scenarios that measure crucial aspects of
a reliable medical system. This paper addresses this gap by providing a
comprehensive evaluation framework for medical question-answering (QA) systems
in a RAG setting for these situations, including sufficiency, integration, and
robustness. We introduce Medical Retrieval-Augmented Generation Benchmark
(MedRGB) that provides various supplementary elements to four medical QA
datasets for testing LLMs' ability to handle these specific scenarios.
Utilizing MedRGB, we conduct extensive evaluations of both state-of-the-art
commercial LLMs and open-source models across multiple retrieval conditions.
Our experimental results reveals current models' limited ability to handle
noise and misinformation in the retrieved documents. We further analyze the
LLMs' reasoning processes to provides valuable insights and future directions
for developing RAG systems in this critical medical domain.
A comprehensive evaluation framework for medical RAG systems addresses reliability, sufficiency, integration, and robustness using a new benchmark with supplementary elements.
retrieval-augmented generation large language models medical domain knowledge-intensive tasks medical question-answering Medical Retrieval-Augmented Generation Benchmark MedRGB