When Models Lie, We Learn: Multilingual Span-Level Hallucination
  Detection with PsiloQA
Hallucination detection remains a fundamental challenge for the safe and
reliable deployment of large language models (LLMs), especially in applications
requiring factual accuracy. Existing hallucination benchmarks often operate at
the sequence level and are limited to English, lacking the fine-grained,
multilingual supervision needed for a comprehensive evaluation. In this work,
we introduce PsiloQA, a large-scale, multilingual dataset annotated with
span-level hallucinations across 14 languages. PsiloQA is constructed through
an automated three-stage pipeline: generating question-answer pairs from
Wikipedia using GPT-4o, eliciting potentially hallucinated answers from diverse
LLMs in a no-context setting, and automatically annotating hallucinated spans
using GPT-4o by comparing against golden answers and retrieved context. We
evaluate a wide range of hallucination detection methods -- including
uncertainty quantification, LLM-based tagging, and fine-tuned encoder models --
and show that encoder-based models achieve the strongest performance across
languages. Furthermore, PsiloQA demonstrates effective cross-lingual
generalization and supports robust knowledge transfer to other benchmarks, all
while being significantly more cost-efficient than human-annotated datasets.
Our dataset and results advance the development of scalable, fine-grained
hallucination detection in multilingual settings.
68ed33b8de1fee572713a5b2
Elisei Rykov
claimed_verified
2025-10-18T04:50:24.609Z
636b867ecde3707d10999b96
https://cdn-avatars.huggingface.co/v1/production/uploads/1667991162387-noauth.png
Rykov Elisei
user
lmeribal
68ed33b8de1fee572713a5b3
Kseniia Petrushina
68ed33b8de1fee572713a5b4
Maksim Savkin
68ed33b8de1fee572713a5b5
Valerii Olisov
68ed33b8de1fee572713a5b6
Artem Vazhentsev
68ed33b8de1fee572713a5b7
Kseniia Titova
68ed33b8de1fee572713a5b8
Alexander Panchenko
68ed33b8de1fee572713a5b9
Vasily Konovalov
claimed_verified
2025-10-18T04:50:22.162Z
622b1f6b9f6139daa8e998ce
/avatars/842719c100a5969be75d04da97333675.svg
Vasily Konovalov
user
Vasily
68ed33b8de1fee572713a5ba
Julia Belikova
claimed_verified
2025-10-18T04:50:26.823Z
659a8f0258608c4044cf7dc5
/avatars/bb213ee4e300fc9143b544f3ecdb88e8.svg
Julia Belikova
user
julia-bel
When Models Lie, We Learn: Multilingual Span-Level Hallucination
  Detection with PsiloQA
Hallucination detection remains a fundamental challenge for the safe and
reliable deployment of large language models (LLMs), especially in applications
requiring factual accuracy. Existing hallucination benchmarks often operate at
the sequence level and are limited to English, lacking the fine-grained,
multilingual supervision needed for a comprehensive evaluation. In this work,
we introduce PsiloQA, a large-scale, multilingual dataset annotated with
span-level hallucinations across 14 languages. PsiloQA is constructed through
an automated three-stage pipeline: generating question-answer pairs from
Wikipedia using GPT-4o, eliciting potentially hallucinated answers from diverse
LLMs in a no-context setting, and automatically annotating hallucinated spans
using GPT-4o by comparing against golden answers and retrieved context. We
evaluate a wide range of hallucination detection methods -- including
uncertainty quantification, LLM-based tagging, and fine-tuned encoder models --
and show that encoder-based models achieve the strongest performance across
languages. Furthermore, PsiloQA demonstrates effective cross-lingual
generalization and supports robust knowledge transfer to other benchmarks, all
while being significantly more cost-efficient than human-annotated datasets.
Our dataset and results advance the development of scalable, fine-grained
hallucination detection in multilingual settings.
PsiloQA, a multilingual dataset with span-level hallucinations, enhances hallucination detection in large language models across 14 languages using an automated pipeline and encoder-based models.
large language models hallucination detection PsiloQA span-level hallucinations multilingual GPT-4o uncertainty quantification LLM-based tagging encoder models cross-lingual generalization knowledge transfer