Less is More: Data Ablation for Reinforcement Fine-Tuning of Large Language Models
Haoru Tan, Xiuzhe Wu, Sitong Wu, Shaofeng Zhang, Yanfeng Chen, Xingwu Sun, Jeanne Shen, Xiaojuan Qi
Reinforcement fine-tuning (RFT) is essential for enhancing the reasoning and generalization capabilities of large language models, but its success heavily relies on the quality of the training data. While data selection has been extensively studied in supervised learning, its role in reinforcement learning, particularly during the RFT stage, remains largely underexplored. In this work, we introduce RFT-DA, a data selection framework designed to identify and prioritize valuable training examples (i.e., assign high RFT-DA scores) while filtering out noisy or detrimental ones, thereby enhancing both the efficiency and effectiveness of reinforcement fine-tuning.At the core of RFT-DA is a sample-level ablation analysis that quantifies the importance of each training example by measuring how its removal affects the final training reward, offering a direct estimate of its contribution to model learning. To ensure scalability, we propose a first-order approximation of the RFT-DA score by backtracking through the optimization process and applying temporal differentiation to the sample-wise influence term, along with a first-order Taylor approximation to adjacent time steps. This yields a lightweight, gradient-based estimator that evaluates the alignment between an individual sample’s gradient and the average gradient direction of all training samples, where a higher degree of alignment implies greater training utility. Extensive experiments demonstrate that RFT-DA consistently improves reward performance and accelerates convergence in reinforcement fine-tuning. Notably, in mathematical reasoning tasks, we only require about 20\% of the data selected based on our data ablation score to achieve more stable training and superior results compared to using the entire dataset. The trained 7B model matches or even outperforms the o1-mini across various benchmarks.