Explaining Sources of Uncertainty in Automated Fact-Checking
Understanding sources of a model's uncertainty regarding its predictions is
crucial for effective human-AI collaboration. Prior work proposes using
numerical uncertainty or hedges ("I'm not sure, but ..."), which do not explain
uncertainty that arises from conflicting evidence, leaving users unable to
resolve disagreements or rely on the output. We introduce CLUE
(Conflict-and-Agreement-aware Language-model Uncertainty Explanations), the
first framework to generate natural language explanations of model uncertainty
by (i) identifying relationships between spans of text that expose
claim-evidence or inter-evidence conflicts and agreements that drive the
model's predictive uncertainty in an unsupervised way, and (ii) generating
explanations via prompting and attention steering that verbalize these critical
interactions. Across three language models and two fact-checking datasets, we
show that CLUE produces explanations that are more faithful to the model's
uncertainty and more consistent with fact-checking decisions than prompting for
uncertainty explanations without span-interaction guidance. Human evaluators
judge our explanations to be more helpful, more informative, less redundant,
and more logically consistent with the input than this baseline. CLUE requires
no fine-tuning or architectural changes, making it plug-and-play for any
white-box language model. By explicitly linking uncertainty to evidence
conflicts, it offers practical support for fact-checking and generalises
readily to other tasks that require reasoning over complex information.
68370a318cd8af122276627d
Jingyi Sun
68370a318cd8af122276627e
Greta Warren
68370a318cd8af122276627f
Irina Shklovski
68370a318cd8af1222766280
Isabelle Augenstein
Explaining Sources of Uncertainty in Automated Fact-Checking
Understanding sources of a model's uncertainty regarding its predictions is
crucial for effective human-AI collaboration. Prior work proposes using
numerical uncertainty or hedges ("I'm not sure, but ..."), which do not explain
uncertainty that arises from conflicting evidence, leaving users unable to
resolve disagreements or rely on the output. We introduce CLUE
(Conflict-and-Agreement-aware Language-model Uncertainty Explanations), the
first framework to generate natural language explanations of model uncertainty
by (i) identifying relationships between spans of text that expose
claim-evidence or inter-evidence conflicts and agreements that drive the
model's predictive uncertainty in an unsupervised way, and (ii) generating
explanations via prompting and attention steering that verbalize these critical
interactions. Across three language models and two fact-checking datasets, we
show that CLUE produces explanations that are more faithful to the model's
uncertainty and more consistent with fact-checking decisions than prompting for
uncertainty explanations without span-interaction guidance. Human evaluators
judge our explanations to be more helpful, more informative, less redundant,
and more logically consistent with the input than this baseline. CLUE requires
no fine-tuning or architectural changes, making it plug-and-play for any
white-box language model. By explicitly linking uncertainty to evidence
conflicts, it offers practical support for fact-checking and generalises
readily to other tasks that require reasoning over complex information.
CLUE generates natural language explanations for a language model's uncertainty by identifying and explaining conflicts and agreements in text spans, enhancing the clarity and helpfulness of explanations in tasks like fact-checking.
knowledge graphs span-interaction guidance claim-evidence inter-evidence conflicts agreements predictive uncertainty prompting attention steering fact-checking datasets white-box language models