Bridging Supervised Learning and Reinforcement Learning in Math
  Reasoning
Reinforcement Learning (RL) has played a central role in the recent surge of
LLMs' math abilities by enabling self-improvement through binary verifier
signals. In contrast, Supervised Learning (SL) is rarely considered for such
verification-driven training, largely due to its heavy reliance on reference
answers and inability to reflect on mistakes. In this work, we challenge the
prevailing notion that self-improvement is exclusive to RL and propose
Negative-aware Fine-Tuning (NFT) -- a supervised approach that enables LLMs to
reflect on their failures and improve autonomously with no external teachers.
In online training, instead of throwing away self-generated negative answers,
NFT constructs an implicit negative policy to model them. This implicit policy
is parameterized with the same positive LLM we target to optimize on positive
data, enabling direct policy optimization on all LLMs' generations. We conduct
experiments on 7B and 32B models in math reasoning tasks. Results consistently
show that through the additional leverage of negative feedback, NFT
significantly improves over SL baselines like Rejection sampling Fine-Tuning,
matching or even surpassing leading RL algorithms like GRPO and DAPO.
Furthermore, we demonstrate that NFT and GRPO are actually equivalent in
strict-on-policy training, even though they originate from entirely different
theoretical foundations. Our experiments and theoretical findings bridge the
gap between SL and RL methods in binary-feedback learning systems.
68360e6c4e92d78afe73c338
Huayu Chen
68360e6c4e92d78afe73c339
Kaiwen Zheng
claimed_verified
2025-05-28T08:58:52.235Z
652bf7edc3cba555d5673c6e
/avatars/78f6416c30203b30671f8423f061c657.svg
Kaiwen Zheng
user
worstcoder
68360e6c4e92d78afe73c33a
Qinsheng Zhang
68360e6c4e92d78afe73c33b
Ganqu Cui
68360e6c4e92d78afe73c33c
Yin Cui
claimed_verified
2025-05-28T19:14:27.483Z
649f05367b57fab3a5b27c8b
https://cdn-avatars.huggingface.co/v1/production/uploads/649f05367b57fab3a5b27c8b/UDJB4yqF2NmaRwCyTOfcl.jpeg
Yin Cui
user
richardaecn
68360e6c4e92d78afe73c33d
Haotian Ye
68360e6c4e92d78afe73c33e
Tsung-Yi Lin
68360e6c4e92d78afe73c33f
Ming-Yu Liu
68360e6c4e92d78afe73c340
Jun Zhu
68360e6c4e92d78afe73c341
Haoxiang Wang
Bridging Supervised Learning and Reinforcement Learning in Math
  Reasoning
Reinforcement Learning (RL) has played a central role in the recent surge of
LLMs' math abilities by enabling self-improvement through binary verifier
signals. In contrast, Supervised Learning (SL) is rarely considered for such
verification-driven training, largely due to its heavy reliance on reference
answers and inability to reflect on mistakes. In this work, we challenge the
prevailing notion that self-improvement is exclusive to RL and propose
Negative-aware Fine-Tuning (NFT) -- a supervised approach that enables LLMs to
reflect on their failures and improve autonomously with no external teachers.
In online training, instead of throwing away self-generated negative answers,
NFT constructs an implicit negative policy to model them. This implicit policy
is parameterized with the same positive LLM we target to optimize on positive
data, enabling direct policy optimization on all LLMs' generations. We conduct
experiments on 7B and 32B models in math reasoning tasks. Results consistently
show that through the additional leverage of negative feedback, NFT
significantly improves over SL baselines like Rejection sampling Fine-Tuning,
matching or even surpassing leading RL algorithms like GRPO and DAPO.
Furthermore, we demonstrate that NFT and GRPO are actually equivalent in
strict-on-policy training, even though they originate from entirely different
theoretical foundations. Our experiments and theoretical findings bridge the
gap between SL and RL methods in binary-feedback learning systems.
Negative-aware Fine-Tuning (NFT) enhances LLMs' math abilities using supervised learning with negative feedback, achieving performance comparable to RL methods.
Reinforcement Learning (RL) Supervised Learning (SL) Negative-aware Fine-Tuning (NFT) binary verifier signals negative answers implicit negative policy policy optimization math reasoning tasks Rejection sampling Fine-Tuning GRPO DAPO strict-on-policy training