KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality
Large Language Models (LLMs), particularly slow-thinking models, often
exhibit severe hallucination, outputting incorrect content due to an inability
to accurately recognize knowledge boundaries during reasoning. While
Reinforcement Learning (RL) can enhance complex reasoning abilities, its
outcome-oriented reward mechanism often lacks factual supervision over the
thinking process, further exacerbating the hallucination problem. To address
the high hallucination in slow-thinking models, we propose Knowledge-enhanced
RL, KnowRL. KnowRL guides models to perform fact-based slow thinking by
integrating a factuality reward, based on knowledge verification, into the RL
training process, helping them recognize their knowledge boundaries. KnowRL
guides models to perform fact-based slow thinking by integrating a factuality
reward, based on knowledge verification, into the RL training process, helping
them recognize their knowledge boundaries. This targeted factual input during
RL training enables the model to learn and internalize fact-based reasoning
strategies. By directly rewarding adherence to facts within the reasoning
steps, KnowRL fosters a more reliable thinking process. Experimental results on
three hallucination evaluation datasets and two reasoning evaluation datasets
demonstrate that KnowRL effectively mitigates hallucinations in slow-thinking
models while maintaining their original strong reasoning capabilities. Our code
is available at https://github.com/zjunlp/KnowRL.
685b75edd2ee4fac76521e7c
Baochang Ren
685b75edd2ee4fac76521e7d
Shuofei Qiao
685b75edd2ee4fac76521e7e
Wenhao Yu
685b75edd2ee4fac76521e7f
Huajun Chen
685b75edd2ee4fac76521e80
Ningyu Zhang
claimed_verified
2025-06-25T11:04:10.718Z
620b3bbb0668e435407c8d0a
/avatars/e0fccbb2577d76088e09f054c35cffbc.svg
Ningyu Zhang
user
Ningyu
KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality
Large Language Models (LLMs), particularly slow-thinking models, often
exhibit severe hallucination, outputting incorrect content due to an inability
to accurately recognize knowledge boundaries during reasoning. While
Reinforcement Learning (RL) can enhance complex reasoning abilities, its
outcome-oriented reward mechanism often lacks factual supervision over the
thinking process, further exacerbating the hallucination problem. To address
the high hallucination in slow-thinking models, we propose Knowledge-enhanced
RL, KnowRL. KnowRL guides models to perform fact-based slow thinking by
integrating a factuality reward, based on knowledge verification, into the RL
training process, helping them recognize their knowledge boundaries. KnowRL
guides models to perform fact-based slow thinking by integrating a factuality
reward, based on knowledge verification, into the RL training process, helping
them recognize their knowledge boundaries. This targeted factual input during
RL training enables the model to learn and internalize fact-based reasoning
strategies. By directly rewarding adherence to facts within the reasoning
steps, KnowRL fosters a more reliable thinking process. Experimental results on
three hallucination evaluation datasets and two reasoning evaluation datasets
demonstrate that KnowRL effectively mitigates hallucinations in slow-thinking
models while maintaining their original strong reasoning capabilities. Our code
is available at https://github.com/zjunlp/KnowRL.
KnowRL, a knowledge-enhanced reinforcement learning approach, reduces hallucinations in slow-thinking large language models by incorporating factuality rewards based on knowledge verification during training.
Large Language Models slow-thinking models hallucination Reinforcement Learning KnowRL factuality reward knowledge verification reasoning reasoning capabilities