On the Generalization of SFT: A Reinforcement Learning Perspective with
  Reward Rectification
We present a simple yet theoretically motivated improvement to Supervised
Fine-Tuning (SFT) for the Large Language Model (LLM), addressing its limited
generalization compared to reinforcement learning (RL). Through mathematical
analysis, we reveal that standard SFT gradients implicitly encode a problematic
reward structure that may severely restrict the generalization capabilities of
model. To rectify this, we propose Dynamic Fine-Tuning (DFT), stabilizing
gradient updates for each token by dynamically rescaling the objective function
with the probability of this token. Remarkably, this single-line code change
significantly outperforms standard SFT across multiple challenging benchmarks
and base models, demonstrating greatly improved generalization. Additionally,
our approach shows competitive results in offline RL settings, offering an
effective yet simpler alternative. This work bridges theoretical insight and
practical solutions, substantially advancing SFT performance. The code will be
available at https://github.com/yongliang-wu/DFT.
6895566648b0ae5ca2710d05
Yongliang Wu
6895566648b0ae5ca2710d06
Yizhou Zhou
6895566648b0ae5ca2710d07
Zhou Ziheng
6895566648b0ae5ca2710d08
Yingzhe Peng
6895566648b0ae5ca2710d09
Xinyu Ye
6895566648b0ae5ca2710d0a
Xinting Hu
6895566648b0ae5ca2710d0b
Wenbo Zhu
6895566648b0ae5ca2710d0c
Lu Qi
6895566648b0ae5ca2710d0d
Ming-Hsuan Yang
6895566648b0ae5ca2710d0e
Xu Yang
On the Generalization of SFT: A Reinforcement Learning Perspective with
  Reward Rectification
We present a simple yet theoretically motivated improvement to Supervised
Fine-Tuning (SFT) for the Large Language Model (LLM), addressing its limited
generalization compared to reinforcement learning (RL). Through mathematical
analysis, we reveal that standard SFT gradients implicitly encode a problematic
reward structure that may severely restrict the generalization capabilities of
model. To rectify this, we propose Dynamic Fine-Tuning (DFT), stabilizing
gradient updates for each token by dynamically rescaling the objective function
with the probability of this token. Remarkably, this single-line code change
significantly outperforms standard SFT across multiple challenging benchmarks
and base models, demonstrating greatly improved generalization. Additionally,
our approach shows competitive results in offline RL settings, offering an
effective yet simpler alternative. This work bridges theoretical insight and
practical solutions, substantially advancing SFT performance. The code will be
available at https://github.com/yongliang-wu/DFT.
Dynamic Fine-Tuning (DFT) improves the generalization of Large Language Models (LLMs) by dynamically rescaling gradients, outperforming standard Supervised Fine-Tuning (SFT) and showing competitive results in offline reinforcement learning.
Supervised Fine-Tuning Large Language Model reinforcement learning gradient updates token probability Dynamic Fine-Tuning offline RL