DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in
  Search-Augmented LLMs
Retrieval Augmented Generation (RAG) is a commonly used approach for
enhancing large language models (LLMs) with relevant and up-to-date
information. However, the retrieved sources can often contain conflicting
information and it remains unclear how models should address such
discrepancies. In this work, we first propose a novel taxonomy of knowledge
conflict types in RAG, along with the desired model behavior for each type. We
then introduce CONFLICTS, a high-quality benchmark with expert annotations of
conflict types in a realistic RAG setting. CONFLICTS is the first benchmark
that enables tracking progress on how models address a wide range of knowledge
conflicts. We conduct extensive experiments on this benchmark, showing that
LLMs often struggle to appropriately resolve conflicts between sources. While
prompting LLMs to explicitly reason about the potential conflict in the
retrieved documents significantly improves the quality and appropriateness of
their responses, substantial room for improvement in future research remains.
6849358142e4f9106973f453
Arie Cattan
6849358142e4f9106973f454
Alon Jacovi
6849358142e4f9106973f455
Ori Ram
claimed_verified
2025-06-12T12:42:38.532Z
60bdd4d485a3ab33829b6222
https://cdn-avatars.huggingface.co/v1/production/uploads/1650532012972-60bdd4d485a3ab33829b6222.jpeg
Ori Ram
user
oriram
6849358142e4f9106973f456
Jonathan Herzig
6849358142e4f9106973f457
Roee Aharoni
6849358142e4f9106973f458
Sasha Goldshtein
6849358142e4f9106973f459
Eran Ofek
6849358142e4f9106973f45a
Idan Szpektor
6849358142e4f9106973f45b
Avi Caciularu
DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in
  Search-Augmented LLMs
Retrieval Augmented Generation (RAG) is a commonly used approach for
enhancing large language models (LLMs) with relevant and up-to-date
information. However, the retrieved sources can often contain conflicting
information and it remains unclear how models should address such
discrepancies. In this work, we first propose a novel taxonomy of knowledge
conflict types in RAG, along with the desired model behavior for each type. We
then introduce CONFLICTS, a high-quality benchmark with expert annotations of
conflict types in a realistic RAG setting. CONFLICTS is the first benchmark
that enables tracking progress on how models address a wide range of knowledge
conflicts. We conduct extensive experiments on this benchmark, showing that
LLMs often struggle to appropriately resolve conflicts between sources. While
prompting LLMs to explicitly reason about the potential conflict in the
retrieved documents significantly improves the quality and appropriateness of
their responses, substantial room for improvement in future research remains.
CONFLICTS, a benchmark for evaluating how LLMs handle knowledge conflicts in RAG, reveals significant challenges in conflict resolution but shows improvement with explicit prompting.
Retrieval Augmented Generation large language models knowledge conflict types benchmark expert annotations prompt reasoning