CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language
  Model Pre-training
Selecting high-quality data for pre-training is crucial in shaping the
downstream task performance of language models. A major challenge lies in
identifying this optimal subset, a problem generally considered intractable,
thus necessitating scalable and effective heuristics. In this work, we propose
a data selection method, CoLoR-Filter (Conditional Loss Reduction Filtering),
which leverages an empirical Bayes-inspired approach to derive a simple and
computationally efficient selection criterion based on the relative loss values
of two auxiliary models.
  In addition to the modeling rationale, we evaluate CoLoR-Filter empirically
on two language modeling tasks: (1) selecting data from C4 for domain
adaptation to evaluation on Books and (2) selecting data from C4 for a suite of
downstream multiple-choice question answering tasks. We demonstrate favorable
scaling both as we subselect more aggressively and using small auxiliary models
to select data for large target models. As one headline result, CoLoR-Filter
data selected using a pair of 150m parameter auxiliary models can train a 1.2b
parameter target model to match a 1.2b parameter model trained on 25b randomly
selected tokens with 25x less data for Books and 11x less data for the
downstream tasks.
  Code: https://github.com/davidbrandfonbrener/color-filter-olmo
  Filtered data:
https://huggingface.co/datasets/davidbrandfonbrener/color-filtered-c4
66716957a3f8ed647dd59dd1
David Brandfonbrener
admin_assigned
2024-06-18T16:29:03.692Z
6314bdae4f283a6af50711da
/avatars/ca3fcc8609b492124b4f97fab29a63f1.svg
David Brandfonbrener
user
davidbrandfonbrener
66716957a3f8ed647dd59dd2
Hanlin Zhang
claimed_verified
2024-12-11T09:24:14.829Z
624054bcc2c17da6a63eb539
/avatars/bf52dc0683b4100733f8696a97696d0e.svg
hlzhang109
user
hlzhang109
66716957a3f8ed647dd59dd3
Andreas Kirsch
admin_assigned
2024-06-18T16:29:24.042Z
635831ee5ddda966186fe6a3
/avatars/17aa4b17a4632af5409a31b23f1ecf34.svg
Andreas Kirsch
user
blackhc
66716957a3f8ed647dd59dd4
Jonathan Richard Schwarz
66716957a3f8ed647dd59dd5
Sham Kakade
CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language
  Model Pre-training
Selecting high-quality data for pre-training is crucial in shaping the
downstream task performance of language models. A major challenge lies in
identifying this optimal subset, a problem generally considered intractable,
thus necessitating scalable and effective heuristics. In this work, we propose
a data selection method, CoLoR-Filter (Conditional Loss Reduction Filtering),
which leverages an empirical Bayes-inspired approach to derive a simple and
computationally efficient selection criterion based on the relative loss values
of two auxiliary models.
  In addition to the modeling rationale, we evaluate CoLoR-Filter empirically
on two language modeling tasks: (1) selecting data from C4 for domain
adaptation to evaluation on Books and (2) selecting data from C4 for a suite of
downstream multiple-choice question answering tasks. We demonstrate favorable
scaling both as we subselect more aggressively and using small auxiliary models
to select data for large target models. As one headline result, CoLoR-Filter
data selected using a pair of 150m parameter auxiliary models can train a 1.2b
parameter target model to match a 1.2b parameter model trained on 25b randomly
selected tokens with 25x less data for Books and 11x less data for the
downstream tasks.
  Code: https://github.com/davidbrandfonbrener/color-filter-olmo
  Filtered data:
https://huggingface.co/datasets/davidbrandfonbrener/color-filtered-c4
A method called CoLoR-Filter uses an empirical Bayes-inspired approach to efficiently select high-quality data for training language models, demonstrating significant reductions in required data compared to random selection.
CoLoR-Filter Conditional Loss Reduction Filtering empirical Bayes-inspired approach language models data selection domain adaptation multiple-choice question answering parameter-efficient fine-tuning