When Less is More: Investigating Data Pruning for Pretraining LLMs at
  Scale
Large volumes of text data have contributed significantly to the development
of large language models (LLMs) in recent years. This data is typically
acquired by scraping the internet, leading to pretraining datasets comprised of
noisy web text. To date, efforts to prune these datasets down to a higher
quality subset have relied on hand-crafted heuristics encoded as rule-based
filters. In this work, we take a wider view and explore scalable estimates of
data quality that can be used to systematically measure the quality of
pretraining data. We perform a rigorous comparison at scale of the simple data
quality estimator of perplexity, as well as more sophisticated and
computationally intensive estimates of the Error L2-Norm and memorization.
These metrics are used to rank and prune pretraining corpora, and we
subsequently compare LLMs trained on these pruned datasets. Surprisingly, we
find that the simple technique of perplexity outperforms our more
computationally expensive scoring methods. We improve over our no-pruning
baseline while training on as little as 30% of the original training dataset.
Our work sets the foundation for unexplored strategies in automatically
curating high quality corpora and suggests the majority of pretraining data can
be removed while retaining performance.
64ffc6723587d3ebfd8041be
Max Marion
2023-09-12T11:02:21.259Z
64ffc6723587d3ebfd8041bf
Ahmet Üstün
admin_assigned
2023-09-12T11:02:36.732Z
60d35d7ad7b174177faabd5b
/avatars/4e5403b9d4a845a2d21e8217dc3c16d2.svg
Ahmet Üstün
user
ahmetu
64ffc6723587d3ebfd8041c0
Luiza Pozzobon
admin_assigned
2023-09-12T11:02:49.169Z
62740039e2be275e63b783e1
https://cdn-avatars.huggingface.co/v1/production/uploads/62740039e2be275e63b783e1/Ft9kE-JS4LPubVUiAgrv0.jpeg
Luiza Pozzobon
user
luizapzbn
64ffc6723587d3ebfd8041c1
Alex Wang
claimed_verified
2023-10-22T19:34:45.263Z
618d96c9b8de35a67a79d288
/avatars/3fbda3277052b5a2d34f84330c440e69.svg
Alex Wang
user
w4ngatang
64ffc6723587d3ebfd8041c2
Marzieh Fadaee
claimed_verified
2024-03-21T12:06:26.345Z
6441042d5d600fb0951a5f99
https://cdn-avatars.huggingface.co/v1/production/uploads/6441042d5d600fb0951a5f99/4CbOaYcEz99BtVAQvnGTn.jpeg
Marzieh Fadaee
user
MarziehFadaee
64ffc6723587d3ebfd8041c3
Sara Hooker
admin_assigned
2023-09-12T11:03:01.922Z
63434eb76f59b79da07dbddf
https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
Sara Hooker
user
sarahooker
When Less is More: Investigating Data Pruning for Pretraining LLMs at
  Scale
Large volumes of text data have contributed significantly to the development
of large language models (LLMs) in recent years. This data is typically
acquired by scraping the internet, leading to pretraining datasets comprised of
noisy web text. To date, efforts to prune these datasets down to a higher
quality subset have relied on hand-crafted heuristics encoded as rule-based
filters. In this work, we take a wider view and explore scalable estimates of
data quality that can be used to systematically measure the quality of
pretraining data. We perform a rigorous comparison at scale of the simple data
quality estimator of perplexity, as well as more sophisticated and
computationally intensive estimates of the Error L2-Norm and memorization.
These metrics are used to rank and prune pretraining corpora, and we
subsequently compare LLMs trained on these pruned datasets. Surprisingly, we
find that the simple technique of perplexity outperforms our more
computationally expensive scoring methods. We improve over our no-pruning
baseline while training on as little as 30% of the original training dataset.
Our work sets the foundation for unexplored strategies in automatically
curating high quality corpora and suggests the majority of pretraining data can
be removed while retaining performance.
Perplexity outperforms more complex methods for data quality estimation in pruning large language model training datasets, resulting in performance improvement with reduced data.
large language models (LLMs) pretraining datasets noisy web text data quality estimator perplexity Error L2-Norm memorization LLMs pruned datasets