Detecting Corpus-Level Knowledge Inconsistencies in Wikipedia with Large
  Language Models
Wikipedia is the largest open knowledge corpus, widely used worldwide and
serving as a key resource for training large language models (LLMs) and
retrieval-augmented generation (RAG) systems. Ensuring its accuracy is
therefore critical. But how accurate is Wikipedia, and how can we improve it?
  We focus on inconsistencies, a specific type of factual inaccuracy, and
introduce the task of corpus-level inconsistency detection. We present CLAIRE,
an agentic system that combines LLM reasoning with retrieval to surface
potentially inconsistent claims along with contextual evidence for human
review. In a user study with experienced Wikipedia editors, 87.5% reported
higher confidence when using CLAIRE, and participants identified 64.7% more
inconsistencies in the same amount of time.
  Combining CLAIRE with human annotation, we contribute WIKICOLLIDE, the first
benchmark of real Wikipedia inconsistencies. Using random sampling with
CLAIRE-assisted analysis, we find that at least 3.3% of English Wikipedia facts
contradict another fact, with inconsistencies propagating into 7.3% of FEVEROUS
and 4.0% of AmbigQA examples. Benchmarking strong baselines on this dataset
reveals substantial headroom: the best fully automated system achieves an AUROC
of only 75.1%.
  Our results show that contradictions are a measurable component of Wikipedia
and that LLM-based systems like CLAIRE can provide a practical tool to help
editors improve knowledge consistency at scale.
68db8bdad2bf1f4b15ec7858
Sina J. Semnani
claimed_verified
2025-09-30T10:17:58.382Z
62ed66743d88d075d72515fe
https://cdn-avatars.huggingface.co/v1/production/uploads/1659725419224-noauth.jpeg
Sina S
user
s-jse
68db8bdad2bf1f4b15ec7859
Jirayu Burapacheep
68db8bdad2bf1f4b15ec785a
Arpandeep Khatua
68db8bdad2bf1f4b15ec785b
Thanawan Atchariyachanvanit
68db8bdad2bf1f4b15ec785c
Zheng Wang
68db8bdad2bf1f4b15ec785d
Monica S. Lam
Detecting Corpus-Level Knowledge Inconsistencies in Wikipedia with Large
  Language Models
Wikipedia is the largest open knowledge corpus, widely used worldwide and
serving as a key resource for training large language models (LLMs) and
retrieval-augmented generation (RAG) systems. Ensuring its accuracy is
therefore critical. But how accurate is Wikipedia, and how can we improve it?
  We focus on inconsistencies, a specific type of factual inaccuracy, and
introduce the task of corpus-level inconsistency detection. We present CLAIRE,
an agentic system that combines LLM reasoning with retrieval to surface
potentially inconsistent claims along with contextual evidence for human
review. In a user study with experienced Wikipedia editors, 87.5% reported
higher confidence when using CLAIRE, and participants identified 64.7% more
inconsistencies in the same amount of time.
  Combining CLAIRE with human annotation, we contribute WIKICOLLIDE, the first
benchmark of real Wikipedia inconsistencies. Using random sampling with
CLAIRE-assisted analysis, we find that at least 3.3% of English Wikipedia facts
contradict another fact, with inconsistencies propagating into 7.3% of FEVEROUS
and 4.0% of AmbigQA examples. Benchmarking strong baselines on this dataset
reveals substantial headroom: the best fully automated system achieves an AUROC
of only 75.1%.
  Our results show that contradictions are a measurable component of Wikipedia
and that LLM-based systems like CLAIRE can provide a practical tool to help
editors improve knowledge consistency at scale.
CLAIRE, an agentic system combining LLM reasoning and retrieval, improves Wikipedia accuracy by detecting inconsistencies, with human editors reporting higher confidence and identifying more issues.
large language models retrieval-augmented generation corpus-level inconsistency detection CLAIRE WIKICOLLIDE AUROC FEVEROUS AmbigQA