Large-Scale Data Selection for Instruction Tuning
Selecting high-quality training data from a larger pool is a crucial step
when instruction-tuning language models, as carefully curated datasets often
produce models that outperform those trained on much larger, noisier datasets.
Automated data selection approaches for instruction-tuning are typically tested
by selecting small datasets (roughly 10k samples) from small pools (100-200k
samples). However, popular deployed instruction-tuned models often train on
hundreds of thousands to millions of samples, subsampled from even larger data
pools. We present a systematic study of how well data selection methods scale
to these settings, selecting up to 2.5M samples from pools of up to 5.8M
samples and evaluating across 7 diverse tasks. We show that many recently
proposed methods fall short of random selection in this setting (while using
more compute), and even decline in performance when given access to larger
pools of data to select over. However, we find that a variant of
representation-based data selection (RDS+), which uses weighted mean pooling of
pretrained LM hidden states, consistently outperforms more complex methods
across all settings tested -- all whilst being more compute-efficient. Our
findings highlight that the scaling properties of proposed automated selection
methods should be more closely examined. We release our code, data, and models
at https://github.com/hamishivi/automated-instruction-selection.
67c67ff6dec55d10cb10fc9e
Hamish Ivison
claimed_verified
2025-03-04T08:40:13.649Z
62608fc2ffe8827cb1d89f9f
https://cdn-avatars.huggingface.co/v1/production/uploads/1654027835241-62608fc2ffe8827cb1d89f9f.png
Hamish Ivison
user
hamishivi
67c67ff6dec55d10cb10fc9f
Muru Zhang
admin_assigned
2025-03-04T11:14:59.402Z
61cc2cf4dcb47bd5ed3cd3b8
https://cdn-avatars.huggingface.co/v1/production/uploads/61cc2cf4dcb47bd5ed3cd3b8/O-tDNhSPZCZQSQG_BaEG9.png
Muru Zhang
user
nanami
67c67ff6dec55d10cb10fca0
Faeze Brahman
admin_assigned
2025-03-04T11:15:05.562Z
65282b8d578679aac7888aec
https://cdn-avatars.huggingface.co/v1/production/uploads/65282b8d578679aac7888aec/dibBkhH-z1c70mJZZxJ7u.jpeg
Faeze Brahman
user
faezeb
67c67ff6dec55d10cb10fca1
Pang Wei Koh
admin_assigned
2025-03-04T11:15:14.558Z
641b4263abfce26bcf7b27de
/avatars/e91b4205e4f74b0dd8c333c23203a924.svg
Pang Wei Koh
user
pangwei
67c67ff6dec55d10cb10fca2
Pradeep Dasigi
admin_assigned
2025-03-04T11:15:20.400Z
6408fcc93461c51cf735a61e
/avatars/619f3653911d111f046a5a6c30fc8319.svg
Pradeep Dasigi
user
pradeepd
Large-Scale Data Selection for Instruction Tuning
Selecting high-quality training data from a larger pool is a crucial step
when instruction-tuning language models, as carefully curated datasets often
produce models that outperform those trained on much larger, noisier datasets.
Automated data selection approaches for instruction-tuning are typically tested
by selecting small datasets (roughly 10k samples) from small pools (100-200k
samples). However, popular deployed instruction-tuned models often train on
hundreds of thousands to millions of samples, subsampled from even larger data
pools. We present a systematic study of how well data selection methods scale
to these settings, selecting up to 2.5M samples from pools of up to 5.8M
samples and evaluating across 7 diverse tasks. We show that many recently
proposed methods fall short of random selection in this setting (while using
more compute), and even decline in performance when given access to larger
pools of data to select over. However, we find that a variant of
representation-based data selection (RDS+), which uses weighted mean pooling of
pretrained LM hidden states, consistently outperforms more complex methods
across all settings tested -- all whilst being more compute-efficient. Our
findings highlight that the scaling properties of proposed automated selection
methods should be more closely examined. We release our code, data, and models
at https://github.com/hamishivi/automated-instruction-selection.
A study on the scalability of data selection methods in instruction-tuning large language models reveals that a simpler, compute-efficient variant of representation-based data selection outperforms more complex methods.
instruction-tuning language models automated data selection representation-based data selection pretrained LM hidden states weighted mean pooling compute-efficiency