A Minimalist Approach to LLM Reasoning: from Rejection Sampling to
  Reinforce
Reinforcement learning (RL) has become a prevailing approach for fine-tuning
large language models (LLMs) on complex reasoning tasks. Among recent methods,
GRPO stands out for its empirical success in training models such as
DeepSeek-R1, yet the sources of its effectiveness remain poorly understood. In
this work, we revisit GRPO from a reinforce-like algorithm perspective and
analyze its core components. Surprisingly, we find that a simple rejection
sampling baseline, RAFT, which trains only on positively rewarded samples,
yields competitive performance than GRPO and PPO. Our ablation studies reveal
that GRPO's main advantage arises from discarding prompts with entirely
incorrect responses, rather than from its reward normalization. Motivated by
this insight, we propose Reinforce-Rej, a minimal extension of policy gradient
that filters both entirely incorrect and entirely correct samples.
Reinforce-Rej improves KL efficiency and stability, serving as a lightweight
yet effective alternative to more complex RL algorithms. We advocate RAFT as a
robust and interpretable baseline, and suggest that future advances should
focus on more principled designs for incorporating negative samples, rather
than relying on them indiscriminately. Our findings provide guidance for future
work in reward-based LLM post-training.
67ff2d4a86e7ad2b4bea1349
Wei Xiong
67ff2d4a86e7ad2b4bea134a
Jiarui Yao
claimed_verified
2025-05-06T08:36:03.515Z
649369b34f0e40ee1a0ed5ba
/avatars/50d0e77883579d5002906c8d29c26ec5.svg
Maxwell Yao
user
MaxwellJryao
67ff2d4a86e7ad2b4bea134b
Yuhui Xu
67ff2d4a86e7ad2b4bea134c
Bo Pang
67ff2d4a86e7ad2b4bea134d
Lei Wang
claimed_verified
2025-06-09T10:13:34.123Z
646def60df618b303b419323
/avatars/97aa761d5255abf230304cfeade87835.svg
Lei Wang
user
demolei
67ff2d4a86e7ad2b4bea134e
Doyen Sahoo
67ff2d4a86e7ad2b4bea134f
Junnan Li
67ff2d4a86e7ad2b4bea1350
Nan Jiang
67ff2d4a86e7ad2b4bea1351
Tong Zhang
67ff2d4a86e7ad2b4bea1352
Caiming Xiong
67ff2d4a86e7ad2b4bea1353
Hanze Dong
A Minimalist Approach to LLM Reasoning: from Rejection Sampling to
  Reinforce
Reinforcement learning (RL) has become a prevailing approach for fine-tuning
large language models (LLMs) on complex reasoning tasks. Among recent methods,
GRPO stands out for its empirical success in training models such as
DeepSeek-R1, yet the sources of its effectiveness remain poorly understood. In
this work, we revisit GRPO from a reinforce-like algorithm perspective and
analyze its core components. Surprisingly, we find that a simple rejection
sampling baseline, RAFT, which trains only on positively rewarded samples,
yields competitive performance than GRPO and PPO. Our ablation studies reveal
that GRPO's main advantage arises from discarding prompts with entirely
incorrect responses, rather than from its reward normalization. Motivated by
this insight, we propose Reinforce-Rej, a minimal extension of policy gradient
that filters both entirely incorrect and entirely correct samples.
Reinforce-Rej improves KL efficiency and stability, serving as a lightweight
yet effective alternative to more complex RL algorithms. We advocate RAFT as a
robust and interpretable baseline, and suggest that future advances should
focus on more principled designs for incorporating negative samples, rather
than relying on them indiscriminately. Our findings provide guidance for future
work in reward-based LLM post-training.
Reinforce-Rej, a minimal extension of policy gradient, outperforms GRPO and PPO in fine-tuning large language models by filtering out both entirely incorrect and correct samples, improving efficiency and stability.
reinforcement learning large language models GRPO DeepSeek-R1 RAFT reject sampling policy gradient Reinforce-Rej KL efficiency stability reward-based post-training