Turk-LettuceDetect: A Hallucination Detection Models for Turkish RAG
  Applications
The widespread adoption of Large Language Models (LLMs) has been hindered by
their tendency to hallucinate, generating plausible but factually incorrect
information. While Retrieval-Augmented Generation (RAG) systems attempt to
address this issue by grounding responses in external knowledge, hallucination
remains a persistent challenge, particularly for morphologically complex,
low-resource languages like Turkish. This paper introduces Turk-LettuceDetect,
the first suite of hallucination detection models specifically designed for
Turkish RAG applications. Building on the LettuceDetect framework, we formulate
hallucination detection as a token-level classification task and fine-tune
three distinct encoder architectures: a Turkish-specific ModernBERT,
TurkEmbed4STS, and multilingual EuroBERT. These models were trained on a
machine-translated version of the RAGTruth benchmark dataset containing 17,790
instances across question answering, data-to-text generation, and summarization
tasks. Our experimental results show that the ModernBERT-based model achieves
an F1-score of 0.7266 on the complete test set, with particularly strong
performance on structured tasks. The models maintain computational efficiency
while supporting long contexts up to 8,192 tokens, making them suitable for
real-time deployment. Comparative analysis reveals that while state-of-the-art
LLMs demonstrate high recall, they suffer from low precision due to
over-generation of hallucinated content, underscoring the necessity of
specialized detection mechanisms. By releasing our models and translated
dataset, this work addresses a critical gap in multilingual NLP and establishes
a foundation for developing more reliable and trustworthy AI applications for
Turkish and other languages.
68d23b0d1ca7156988a8ee53
Selva Taş
claimed_verified
2025-09-23T10:04:54.670Z
63241e8f0fc33e3d14dd5277
https://cdn-avatars.huggingface.co/v1/production/uploads/63241e8f0fc33e3d14dd5277/QUF4j7L3xZdKRnLRCrA12.jpeg
SELVA TAŞ
user
selvatas
68d23b0d1ca7156988a8ee54
Mahmut El Huseyni
claimed_verified
2025-09-23T10:04:59.202Z
6422eab8e2029ade06eeee2c
https://cdn-avatars.huggingface.co/v1/production/uploads/6422eab8e2029ade06eeee2c/Gai3BHr2WJ0YuhdumqQ_z.png
Mahmud ElHuseyni 🇵🇸
user
MElHuseyni
68d23b0d1ca7156988a8ee55
Özay Ezerceli
claimed_verified
2025-09-23T11:22:30.533Z
6464e76894327a238f56d7ff
https://cdn-avatars.huggingface.co/v1/production/uploads/6464e76894327a238f56d7ff/pKsurCt64lH_0pb_nvjoE.jpeg
Özay Ezerceli
user
ozayezerceli
68d23b0d1ca7156988a8ee56
Reyhan Bayraktar
claimed_verified
2025-09-24T13:54:31.566Z
64a52413fa840dd18c2f76fe
https://cdn-avatars.huggingface.co/v1/production/uploads/64a52413fa840dd18c2f76fe/5PD-BHnKm2aQWFZhE7P2I.jpeg
Reyhan Bayraktar
user
byrayhana
68d23b0d1ca7156988a8ee57
Fatma Betül Terzioğlu
claimed_verified
2025-09-26T12:48:06.593Z
6879146c99e4f3995ef34ba1
https://cdn-avatars.huggingface.co/v1/production/uploads/6879146c99e4f3995ef34ba1/hIMI-6mopkhA8rmLB1Ij6.jpeg
Betül Terzioğlu
user
BetulT
Turk-LettuceDetect: A Hallucination Detection Models for Turkish RAG
  Applications
The widespread adoption of Large Language Models (LLMs) has been hindered by
their tendency to hallucinate, generating plausible but factually incorrect
information. While Retrieval-Augmented Generation (RAG) systems attempt to
address this issue by grounding responses in external knowledge, hallucination
remains a persistent challenge, particularly for morphologically complex,
low-resource languages like Turkish. This paper introduces Turk-LettuceDetect,
the first suite of hallucination detection models specifically designed for
Turkish RAG applications. Building on the LettuceDetect framework, we formulate
hallucination detection as a token-level classification task and fine-tune
three distinct encoder architectures: a Turkish-specific ModernBERT,
TurkEmbed4STS, and multilingual EuroBERT. These models were trained on a
machine-translated version of the RAGTruth benchmark dataset containing 17,790
instances across question answering, data-to-text generation, and summarization
tasks. Our experimental results show that the ModernBERT-based model achieves
an F1-score of 0.7266 on the complete test set, with particularly strong
performance on structured tasks. The models maintain computational efficiency
while supporting long contexts up to 8,192 tokens, making them suitable for
real-time deployment. Comparative analysis reveals that while state-of-the-art
LLMs demonstrate high recall, they suffer from low precision due to
over-generation of hallucinated content, underscoring the necessity of
specialized detection mechanisms. By releasing our models and translated
dataset, this work addresses a critical gap in multilingual NLP and establishes
a foundation for developing more reliable and trustworthy AI applications for
Turkish and other languages.
Turk-LettuceDetect, a suite of hallucination detection models for Turkish RAG applications, achieves high performance using fine-tuned encoder architectures on a machine-translated RAGTruth dataset.
Large Language Models (LLMs) Retrieval-Augmented Generation (RAG) hallucination detection token-level classification ModernBERT TurkEmbed4STS EuroBERT RAGTruth benchmark dataset F1-score computational efficiency long contexts multilingual NLP