The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, Yu Meng
Reinforcement learning with verifiable rewards (RLVR) is a promising approach for training language models (LMs) on reasoning tasks that elicit emergent long chains of thought (CoTs).Unlike supervised learning, it updates the model using both correct and incorrect samples via policy gradients.To better understand its mechanism, we decompose the learning signal into positive and negative sample reinforcement.Surprisingly, we find that training with only negative samples---without reinforcing correct responses---can be highly effective: it consistently improves performance over the base model across the entire Pass@$k$ spectrum, often matching or surpassing PPO and GRPO.In contrast, reinforcing only correct responses improves Pass@$1$ but degrades performance at higher $k$, due to reduced diversity.These inference-scaling trends highlight the critical yet underappreciated role of negative sample reinforcement (NSR) in improving LM reasoning. Through gradient analysis, we show that it works by suppressing incorrect generations and redistributing probability mass toward other plausible candidates, guided by the model's prior beliefs.It refines the model's existing knowledge, without learning new behaviors from scratch. Building on this insight, we propose a simple variant of the RL objective that upweights NSR, and show that it consistently improves overall Pass@$k$ performance on MATH, AIME 2025, and AMC23.