RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented
  Generation Systems
Retrieval-Augmented Generation (RAG) enhances recency and factuality in
answers. However, existing evaluations rarely test how well these systems cope
with real-world noise, conflicting between internal and external retrieved
contexts, or fast-changing facts. We introduce Retrieval-Aware Robustness
Evaluation (RARE), a unified framework and large-scale benchmark that jointly
stress-tests query and document perturbations over dynamic, time-sensitive
corpora. One of the central features of RARE is a knowledge-graph-driven
synthesis pipeline (RARE-Get) that automatically extracts single and multi-hop
relations from the customized corpus and generates multi-level question sets
without manual intervention. Leveraging this pipeline, we construct a dataset
(RARE-Set) spanning 400 expert-level time-sensitive finance, economics, and
policy documents and 48,322 questions whose distribution evolves as the
underlying sources change. To quantify resilience, we formalize
retrieval-conditioned robustness metrics (RARE-Met) that capture a model's
ability to remain correct or recover when queries, documents, or real-world
retrieval results are systematically altered. Our results show that RAG systems
exhibit surprising vulnerability to perturbations, with document robustness
consistently being the weakest point regardless of generator size or
architecture. RAG systems consistently show lower robustness on multi-hop
queries than single-hop queries across all domains.
683f23ffc7f66ac1b46b2e49
Yixiao Zeng
claimed_verified
2025-06-04T09:01:28.771Z
64992b16406dbbb2c592634a
/avatars/7d2e8f10f948db1974ca2d73e1834742.svg
Yixiao Zeng
user
Rabinovich
683f23ffc7f66ac1b46b2e4a
Tianyu Cao
claimed_verified
2025-06-04T09:01:25.943Z
66cf5e9fda5d6bbe4d4755f3
/avatars/407698973be8e12ef4fdf3acddd2705f.svg
Tiya Cao
user
TiyaC
683f23ffc7f66ac1b46b2e4b
Danqing Wang
683f23ffc7f66ac1b46b2e4c
Xinran Zhao
683f23ffc7f66ac1b46b2e4d
Zimeng Qiu
683f23ffc7f66ac1b46b2e4e
Morteza Ziyadi
extracted_confirmed
2025-06-03T16:37:16.891Z
6462db3052193f295764f95f
/avatars/fded932b65d6f7f2773fb2faecde122a.svg
Morteza Ziyadi
user
mziyadi
683f23ffc7f66ac1b46b2e4f
Tongshuang Wu
683f23ffc7f66ac1b46b2e50
Lei Li
RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented
  Generation Systems
Retrieval-Augmented Generation (RAG) enhances recency and factuality in
answers. However, existing evaluations rarely test how well these systems cope
with real-world noise, conflicting between internal and external retrieved
contexts, or fast-changing facts. We introduce Retrieval-Aware Robustness
Evaluation (RARE), a unified framework and large-scale benchmark that jointly
stress-tests query and document perturbations over dynamic, time-sensitive
corpora. One of the central features of RARE is a knowledge-graph-driven
synthesis pipeline (RARE-Get) that automatically extracts single and multi-hop
relations from the customized corpus and generates multi-level question sets
without manual intervention. Leveraging this pipeline, we construct a dataset
(RARE-Set) spanning 400 expert-level time-sensitive finance, economics, and
policy documents and 48,322 questions whose distribution evolves as the
underlying sources change. To quantify resilience, we formalize
retrieval-conditioned robustness metrics (RARE-Met) that capture a model's
ability to remain correct or recover when queries, documents, or real-world
retrieval results are systematically altered. Our results show that RAG systems
exhibit surprising vulnerability to perturbations, with document robustness
consistently being the weakest point regardless of generator size or
architecture. RAG systems consistently show lower robustness on multi-hop
queries than single-hop queries across all domains.
RARE evaluates the robustness of Retrieval-Augmented Generation (RAG) systems against real-world noise, context conflicts, and time-sensitive data with a knowledge-graph-driven benchmark.
Retrieval-Augmented Generation RAG Retrieval-Aware Robustness Evaluation RARE query perturbations document perturbations dynamic corpora knowledge-graph-driven synthesis pipeline RARE-Get RARE-Set retrieval-conditioned robustness metrics RARE-Met multi-hop queries single-hop queries