Selective Self-to-Supervised Fine-Tuning for Generalization in Large
  Language Models
Fine-tuning Large Language Models (LLMs) on specific datasets is a common
practice to improve performance on target tasks. However, this performance gain
often leads to overfitting, where the model becomes too specialized in either
the task or the characteristics of the training data, resulting in a loss of
generalization. This paper introduces Selective Self-to-Supervised Fine-Tuning
(S3FT), a fine-tuning approach that achieves better performance than the
standard supervised fine-tuning (SFT) while improving generalization. S3FT
leverages the existence of multiple valid responses to a query. By utilizing
the model's correct responses, S3FT reduces model specialization during the
fine-tuning stage. S3FT first identifies the correct model responses from the
training set by deploying an appropriate judge. Then, it fine-tunes the model
using the correct model responses and the gold response (or its paraphrase) for
the remaining samples. The effectiveness of S3FT is demonstrated through
experiments on mathematical reasoning, Python programming and reading
comprehension tasks. The results show that standard SFT can lead to an average
performance drop of up to 4.4 on multiple benchmarks, such as MMLU and
TruthfulQA. In contrast, S3FT reduces this drop by half, i.e. 2.5, indicating
better generalization capabilities than SFT while performing significantly
better on the fine-tuning tasks.
67b3716bab1b992c7f4599da
Sonam Gupta
67b3716bab1b992c7f4599db
Yatin Nandwani
admin_assigned
2025-02-19T15:52:54.491Z
6626284411772517e546d799
/avatars/a9f2f1958c9f17ccc911fa323d9af2f3.svg
Yatin Nandwani
user
ynandwan
67b3716bab1b992c7f4599dc
Asaf Yehudai
admin_assigned
2025-02-19T15:53:01.485Z
638324f862badff43269e588
/avatars/907a39a9b44fc8b7f3fad35858b01fb7.svg
Asaf Yehudai
user
Asaf-Yehudai
67b3716bab1b992c7f4599dd
Dinesh Khandelwal
67b3716bab1b992c7f4599de
Dinesh Raghu
67b3716bab1b992c7f4599df
Sachindra Joshi
Selective Self-to-Supervised Fine-Tuning for Generalization in Large
  Language Models
Fine-tuning Large Language Models (LLMs) on specific datasets is a common
practice to improve performance on target tasks. However, this performance gain
often leads to overfitting, where the model becomes too specialized in either
the task or the characteristics of the training data, resulting in a loss of
generalization. This paper introduces Selective Self-to-Supervised Fine-Tuning
(S3FT), a fine-tuning approach that achieves better performance than the
standard supervised fine-tuning (SFT) while improving generalization. S3FT
leverages the existence of multiple valid responses to a query. By utilizing
the model's correct responses, S3FT reduces model specialization during the
fine-tuning stage. S3FT first identifies the correct model responses from the
training set by deploying an appropriate judge. Then, it fine-tunes the model
using the correct model responses and the gold response (or its paraphrase) for
the remaining samples. The effectiveness of S3FT is demonstrated through
experiments on mathematical reasoning, Python programming and reading
comprehension tasks. The results show that standard SFT can lead to an average
performance drop of up to 4.4 on multiple benchmarks, such as MMLU and
TruthfulQA. In contrast, S3FT reduces this drop by half, i.e. 2.5, indicating
better generalization capabilities than SFT while performing significantly
better on the fine-tuning tasks.
Selective Self-to-Supervised Fine-Tuning (S3FT) improves generalization and performance of fine-tuned Large Language Models (LLMs) by leveraging correct responses and reducing specialization.
Fine-Tuning Large Language Models (LLMs) overfitting Selective Self-to-Supervised Fine-Tuning (S3FT) standard supervised fine-tuning (SFT) generalization mathematical reasoning Python programming reading comprehension MMLU TruthfulQA