Learning to Reason for Factuality
Reasoning Large Language Models (R-LLMs) have significantly advanced complex
reasoning tasks but often struggle with factuality, generating substantially
more hallucinations than their non-reasoning counterparts on long-form
factuality benchmarks. However, extending online Reinforcement Learning (RL), a
key component in recent R-LLM advancements, to the long-form factuality setting
poses several unique challenges due to the lack of reliable verification
methods. Previous work has utilized automatic factuality evaluation frameworks
such as FActScore to curate preference data in the offline RL setting, yet we
find that directly leveraging such methods as the reward in online RL leads to
reward hacking in multiple ways, such as producing less detailed or relevant
responses. We propose a novel reward function that simultaneously considers the
factual precision, response detail level, and answer relevance, and applies
online RL to learn high quality factual reasoning. Evaluated on six long-form
factuality benchmarks, our factual reasoning model achieves an average
reduction of 23.1 percentage points in hallucination rate, a 23% increase in
answer detail level, and no degradation in the overall response helpfulness.
6895986548b0ae5ca2710e2b
Xilun Chen
6895986548b0ae5ca2710e2c
Ilia Kulikov
6895986548b0ae5ca2710e2d
Vincent-Pierre Berges
6895986548b0ae5ca2710e2e
Barlas Oğuz
6895986548b0ae5ca2710e2f
Rulin Shao
6895986548b0ae5ca2710e30
Gargi Ghosh
6895986548b0ae5ca2710e31
Jason Weston
6895986548b0ae5ca2710e32
Wen-tau Yih
Learning to Reason for Factuality
Reasoning Large Language Models (R-LLMs) have significantly advanced complex
reasoning tasks but often struggle with factuality, generating substantially
more hallucinations than their non-reasoning counterparts on long-form
factuality benchmarks. However, extending online Reinforcement Learning (RL), a
key component in recent R-LLM advancements, to the long-form factuality setting
poses several unique challenges due to the lack of reliable verification
methods. Previous work has utilized automatic factuality evaluation frameworks
such as FActScore to curate preference data in the offline RL setting, yet we
find that directly leveraging such methods as the reward in online RL leads to
reward hacking in multiple ways, such as producing less detailed or relevant
responses. We propose a novel reward function that simultaneously considers the
factual precision, response detail level, and answer relevance, and applies
online RL to learn high quality factual reasoning. Evaluated on six long-form
factuality benchmarks, our factual reasoning model achieves an average
reduction of 23.1 percentage points in hallucination rate, a 23% increase in
answer detail level, and no degradation in the overall response helpfulness.
A novel reward function for online reinforcement learning improves factuality and detail in reasoning large language models without reducing helpfulness.
Reasoning Large Language Models R-LLMs Reinforcement Learning RL online RL offline RL FActScore reward hacking factual precision answer relevance