Rethinking Data Selection at Scale: Random Selection is Almost All You
  Need
Supervised fine-tuning (SFT) is crucial for aligning Large Language Models
(LLMs) with human instructions. The primary goal during SFT is to select a
small yet representative subset of training data from the larger pool, such
that fine-tuning with this subset achieves results comparable to or even
exceeding those obtained using the entire dataset. However, most existing data
selection techniques are designed for small-scale data pools, which fail to
meet the demands of real-world SFT scenarios. In this paper, we replicated
several self-scoring methods those that do not rely on external model
assistance on two million scale datasets, and found that nearly all methods
struggled to significantly outperform random selection when dealing with such
large-scale data pools. Moreover, our comparisons suggest that, during SFT,
diversity in data selection is more critical than simply focusing on high
quality data. We also analyzed the limitations of several current approaches,
explaining why they perform poorly on large-scale datasets and why they are
unsuitable for such contexts. Finally, we found that filtering data by token
length offers a stable and efficient method for improving results. This
approach, particularly when training on long text data, proves highly
beneficial for relatively weaker base models, such as Llama3.
670dd0cc487a5e24b27713b9
Tingyu Xia
claimed_verified
2024-10-16T09:44:22.078Z
653b5cb0d994e992e2831539
/avatars/e6b917cb263f10a6d9c72ec539276428.svg
xiatingyu
user
xiatingyu
670dd0cc487a5e24b27713ba
Bowen Yu
admin_assigned
2024-10-15T14:21:17.898Z
6438b43ab2ea24b52ebac2b9
/avatars/84133cd719a4b1e2f5c1a74178425f86.svg
Bowen Yu
user
bwy
670dd0cc487a5e24b27713bb
Kai Dang
670dd0cc487a5e24b27713bc
An Yang
claimed_verified
2025-05-19T06:43:16.208Z
62088594a5943c8a8fc94560
https://cdn-avatars.huggingface.co/v1/production/uploads/1644733028938-62088594a5943c8a8fc94560.png
An Yang
user
yangapku
670dd0cc487a5e24b27713bd
Yuan Wu
claimed_verified
2024-10-15T15:17:56.143Z
670e57b3391f1a7021182bff
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/N0tuHZVz8KFPCv8G1qUX2.png
Yuan Wu
user
WhiteCatY
670dd0cc487a5e24b27713be
Yuan Tian
670dd0cc487a5e24b27713bf
Yi Chang
670dd0cc487a5e24b27713c0
Junyang Lin
admin_assigned
2024-10-15T14:19:07.666Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
Rethinking Data Selection at Scale: Random Selection is Almost All You
  Need
Supervised fine-tuning (SFT) is crucial for aligning Large Language Models
(LLMs) with human instructions. The primary goal during SFT is to select a
small yet representative subset of training data from the larger pool, such
that fine-tuning with this subset achieves results comparable to or even
exceeding those obtained using the entire dataset. However, most existing data
selection techniques are designed for small-scale data pools, which fail to
meet the demands of real-world SFT scenarios. In this paper, we replicated
several self-scoring methods those that do not rely on external model
assistance on two million scale datasets, and found that nearly all methods
struggled to significantly outperform random selection when dealing with such
large-scale data pools. Moreover, our comparisons suggest that, during SFT,
diversity in data selection is more critical than simply focusing on high
quality data. We also analyzed the limitations of several current approaches,
explaining why they perform poorly on large-scale datasets and why they are
unsuitable for such contexts. Finally, we found that filtering data by token
length offers a stable and efficient method for improving results. This
approach, particularly when training on long text data, proves highly
beneficial for relatively weaker base models, such as Llama3.
Selecting representative subsets for supervised fine-tuning of LLMs from large datasets is challenging, and diversity in data selection is more critical than high data quality; token length filtering improves results.
Large Language Models Supervised fine-tuning self-scoring methods data selection large-scale datasets diversity base models token length filtering Llama3