RadEval: A framework for radiology text evaluation
We introduce RadEval, a unified, open-source framework for evaluating
radiology texts. RadEval consolidates a diverse range of metrics, from classic
n-gram overlap (BLEU, ROUGE) and contextual measures (BERTScore) to clinical
concept-based scores (F1CheXbert, F1RadGraph, RaTEScore, SRR-BERT,
TemporalEntityF1) and advanced LLM-based evaluators (GREEN). We refine and
standardize implementations, extend GREEN to support multiple imaging
modalities with a more lightweight model, and pretrain a domain-specific
radiology encoder, demonstrating strong zero-shot retrieval performance. We
also release a richly annotated expert dataset with over 450 clinically
significant error labels and show how different metrics correlate with
radiologist judgment. Finally, RadEval provides statistical testing tools and
baseline model evaluations across multiple publicly available datasets,
facilitating reproducibility and robust benchmarking in radiology report
generation.
68d40c58e151e844b496c5fb
Justin Xu
68d40c58e151e844b496c5fc
Xi Zhang
claimed_verified
2025-09-25T07:11:27.707Z
652ebdcc76365388909b06cf
https://cdn-avatars.huggingface.co/v1/production/uploads/652ebdcc76365388909b06cf/MnIVVuTphX8buEER9EkCY.jpeg
Xi Zhang
user
X-iZhang
68d40c58e151e844b496c5fd
Javid Abderezaei
68d40c58e151e844b496c5fe
Julie Bauml
68d40c58e151e844b496c5ff
Roger Boodoo
68d40c58e151e844b496c600
Fatemeh Haghighi
68d40c58e151e844b496c601
Ali Ganjizadeh
68d40c58e151e844b496c602
Eric Brattain
68d40c58e151e844b496c603
Dave Van Veen
68d40c58e151e844b496c604
Zaiqiao Meng
68d40c58e151e844b496c605
David Eyre
68d40c58e151e844b496c606
Jean-Benoit Delbrouck
RadEval: A framework for radiology text evaluation
We introduce RadEval, a unified, open-source framework for evaluating
radiology texts. RadEval consolidates a diverse range of metrics, from classic
n-gram overlap (BLEU, ROUGE) and contextual measures (BERTScore) to clinical
concept-based scores (F1CheXbert, F1RadGraph, RaTEScore, SRR-BERT,
TemporalEntityF1) and advanced LLM-based evaluators (GREEN). We refine and
standardize implementations, extend GREEN to support multiple imaging
modalities with a more lightweight model, and pretrain a domain-specific
radiology encoder, demonstrating strong zero-shot retrieval performance. We
also release a richly annotated expert dataset with over 450 clinically
significant error labels and show how different metrics correlate with
radiologist judgment. Finally, RadEval provides statistical testing tools and
baseline model evaluations across multiple publicly available datasets,
facilitating reproducibility and robust benchmarking in radiology report
generation.
RadEval is a comprehensive framework for evaluating radiology texts using a variety of metrics, including n-gram overlap, contextual measures, clinical concept-based scores, and advanced LLM-based evaluators, with a focus on reproducibility and robust benchmarking.
n-gram overlap BLEU ROUGE contextual measures BERTScore clinical concept-based scores F1CheXbert F1RadGraph RaTEScore SRR-BERT TemporalEntityF1 advanced LLM-based evaluators GREEN zero-shot retrieval radiology encoder expert dataset radiologist judgment statistical testing tools baseline model evaluations