TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning
While large language models (LLMs) have demonstrated strong performance on
factoid question answering, they are still prone to hallucination and
untruthful responses, particularly when tasks demand information outside their
parametric knowledge. Indeed, truthfulness requires more than accuracy --
models must also recognize uncertainty and abstain when unsure to avoid
hallucinations. This presents a fundamental challenge for existing methods:
approaches that optimize for accuracy often amplify hallucinations, while those
that encourage abstention can become overly conservative, sacrificing correct
answers. Both extremes ultimately compromise truthfulness. In this work, we
present TruthRL, a general reinforcement learning (RL) framework that directly
optimizes the truthfulness of LLMs. Specifically, we implement TruthRL using
GRPO with a simple yet effective ternary reward that distinguishes correct
answers, hallucinations, and abstentions. It incentivizes models to reduce
hallucinations not only by providing correct responses, but also by enabling
abstention when uncertain, thereby improving truthfulness. Extensive
experiments across four knowledge-intensive benchmarks show that, compared to
vanilla RL, TruthRL significantly reduces hallucinations by 28.9% and improves
truthfulness by 21.1%, with consistent gains across various backbone models
(e.g., Qwen, Llama) under both retrieval and non-retrieval setups. In-depth
ablation study demonstrates that vanilla accuracy-driven methods, such as
supervised fine-tuning or RL with a binary reward, struggle to balance factual
correctness and uncertainty. In contrast, our proposed truthfulness-driven
TruthRL achieves strong performance in both accuracy and truthfulness,
underscoring the importance of learning objective design for developing
truthful LLMs.
68dc90b84159d1f2418f9aa7
Zhepei Wei
68dc90b84159d1f2418f9aa8
Xiao Yang
68dc90b84159d1f2418f9aa9
Kai Sun
68dc90b84159d1f2418f9aaa
Jiaqi Wang
68dc90b84159d1f2418f9aab
Rulin Shao
68dc90b84159d1f2418f9aac
Sean Chen
68dc90b84159d1f2418f9aad
Mohammad Kachuee
68dc90b84159d1f2418f9aae
Teja Gollapudi
claimed_verified
2025-10-01T10:21:59.364Z
62bcdae6e75a73c22a18b031
https://cdn-avatars.huggingface.co/v1/production/uploads/62bcdae6e75a73c22a18b031/2jkNvSxIzXtpfMNfFaj9i.png
Teja Gollapudi
user
Teja-Gollapudi
68dc90b84159d1f2418f9aaf
Tony Liao
68dc90b84159d1f2418f9ab0
Nicolas Scheffer
68dc90b84159d1f2418f9ab1
Rakesh Wanga
68dc90b84159d1f2418f9ab2
Anuj Kumar
68dc90b84159d1f2418f9ab3
Yu Meng
68dc90b84159d1f2418f9ab4
Wen-tau Yih
68dc90b84159d1f2418f9ab5
Xin Luna Dong
TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning
While large language models (LLMs) have demonstrated strong performance on
factoid question answering, they are still prone to hallucination and
untruthful responses, particularly when tasks demand information outside their
parametric knowledge. Indeed, truthfulness requires more than accuracy --
models must also recognize uncertainty and abstain when unsure to avoid
hallucinations. This presents a fundamental challenge for existing methods:
approaches that optimize for accuracy often amplify hallucinations, while those
that encourage abstention can become overly conservative, sacrificing correct
answers. Both extremes ultimately compromise truthfulness. In this work, we
present TruthRL, a general reinforcement learning (RL) framework that directly
optimizes the truthfulness of LLMs. Specifically, we implement TruthRL using
GRPO with a simple yet effective ternary reward that distinguishes correct
answers, hallucinations, and abstentions. It incentivizes models to reduce
hallucinations not only by providing correct responses, but also by enabling
abstention when uncertain, thereby improving truthfulness. Extensive
experiments across four knowledge-intensive benchmarks show that, compared to
vanilla RL, TruthRL significantly reduces hallucinations by 28.9% and improves
truthfulness by 21.1%, with consistent gains across various backbone models
(e.g., Qwen, Llama) under both retrieval and non-retrieval setups. In-depth
ablation study demonstrates that vanilla accuracy-driven methods, such as
supervised fine-tuning or RL with a binary reward, struggle to balance factual
correctness and uncertainty. In contrast, our proposed truthfulness-driven
TruthRL achieves strong performance in both accuracy and truthfulness,
underscoring the importance of learning objective design for developing
truthful LLMs.
TruthRL, a reinforcement learning framework, enhances the truthfulness of large language models by balancing accuracy and abstention, significantly reducing hallucinations and improving performance across benchmarks.
large language models LLMs hallucination untruthful responses parametric knowledge truthfulness reinforcement learning RL GRPO ternary reward abstention accuracy-driven methods supervised fine-tuning binary reward knowledge-intensive benchmarks Qwen Llama retrieval non-retrieval setups ablation study