The Hallucination Tax of Reinforcement Finetuning
Reinforcement finetuning (RFT) has become a standard approach for enhancing
the reasoning capabilities of large language models (LLMs). However, its impact
on model trustworthiness remains underexplored. In this work, we identify and
systematically study a critical side effect of RFT, which we term the
hallucination tax: a degradation in refusal behavior causing models to produce
hallucinated answers to unanswerable questions confidently. To investigate
this, we introduce SUM (Synthetic Unanswerable Math), a high-quality dataset of
unanswerable math problems designed to probe models' ability to recognize an
unanswerable question by reasoning from the insufficient or ambiguous
information. Our results show that standard RFT training could reduce model
refusal rates by more than 80%, which significantly increases model's tendency
to hallucinate. We further demonstrate that incorporating just 10% SUM during
RFT substantially restores appropriate refusal behavior, with minimal accuracy
trade-offs on solvable tasks. Crucially, this approach enables LLMs to leverage
inference-time compute to reason about their own uncertainty and knowledge
boundaries, improving generalization not only to out-of-domain math problems
but also to factual question answering tasks.
682d355b89b5f79a1d6546f9
Linxin Song
claimed_verified
2025-06-07T13:34:02.506Z
64d660308ebc40443813f014
/avatars/516bb2d2383be99794e366dfb41636b6.svg
Linxin Song
user
linxinso
682d355b89b5f79a1d6546fa
Taiwei Shi
claimed_verified
2025-05-22T07:17:53.968Z
62e1b3cb3eb0730f621a83f6
https://cdn-avatars.huggingface.co/v1/production/uploads/1658958764563-noauth.jpeg
Taiwei Shi
user
MaksimSTW
682d355b89b5f79a1d6546fb
Jieyu Zhao
The Hallucination Tax of Reinforcement Finetuning
Reinforcement finetuning (RFT) has become a standard approach for enhancing
the reasoning capabilities of large language models (LLMs). However, its impact
on model trustworthiness remains underexplored. In this work, we identify and
systematically study a critical side effect of RFT, which we term the
hallucination tax: a degradation in refusal behavior causing models to produce
hallucinated answers to unanswerable questions confidently. To investigate
this, we introduce SUM (Synthetic Unanswerable Math), a high-quality dataset of
unanswerable math problems designed to probe models' ability to recognize an
unanswerable question by reasoning from the insufficient or ambiguous
information. Our results show that standard RFT training could reduce model
refusal rates by more than 80%, which significantly increases model's tendency
to hallucinate. We further demonstrate that incorporating just 10% SUM during
RFT substantially restores appropriate refusal behavior, with minimal accuracy
trade-offs on solvable tasks. Crucially, this approach enables LLMs to leverage
inference-time compute to reason about their own uncertainty and knowledge
boundaries, improving generalization not only to out-of-domain math problems
but also to factual question answering tasks.
Reinforcement fine-tuning can degrade model refusal behavior, leading to increased hallucination; incorporating a synthetic dataset of unanswerable math problems during fine-tuning can restore appropriate refusal behavior with minimal accuracy loss and improve generalization.
Reinforcement finetuning (RFT) large language models (LLMs) hallucination tax refusal behavior SUM (Synthetic Unanswerable Math) unanswerable math problems solvable tasks inference-time compute knowledge boundaries generalization factual question answering tasks