What's In My Big Data?
Large text corpora are the backbone of language models. However, we have a
limited understanding of the content of these corpora, including general
statistics, quality, social factors, and inclusion of evaluation data
(contamination). In this work, we propose What's In My Big Data? (WIMBD), a
platform and a set of sixteen analyses that allow us to reveal and compare the
contents of large text corpora. WIMBD builds on two basic capabilities -- count
and search -- at scale, which allows us to analyze more than 35 terabytes on a
standard compute node. We apply WIMBD to ten different corpora used to train
popular language models, including C4, The Pile, and RedPajama. Our analysis
uncovers several surprising and previously undocumented findings about these
corpora, including the high prevalence of duplicate, synthetic, and low-quality
content, personally identifiable information, toxic language, and benchmark
contamination. For instance, we find that about 50% of the documents in
RedPajama and LAION-2B-en are duplicates. In addition, several datasets used
for benchmarking models trained on such corpora are contaminated with respect
to important benchmarks, including the Winograd Schema Challenge and parts of
GLUE and SuperGLUE. We open-source WIMBD's code and artifacts to provide a
standard set of evaluations for new text-based corpora and to encourage more
analyses and transparency around them: github.com/allenai/wimbd.
6541c93cea69bcb62029a415
Yanai Elazar
extracted_confirmed
2023-11-01T04:57:17.205Z
623ca115a795593324c4353f
/avatars/bf11fe728df2786d52ed4d2de12b48d3.svg
Yanai Elazar
user
yanaiela
6541c93cea69bcb62029a416
Akshita Bhagia
claimed_verified
2023-12-19T19:21:41.372Z
6282937a9624ab5f75c977e1
https://cdn-avatars.huggingface.co/v1/production/uploads/6282937a9624ab5f75c977e1/ZOysQbGzEzmrN_SaTQgOP.jpeg
Akshita Bhagia
user
akshitab
6541c93cea69bcb62029a417
Ian Magnusson
6541c93cea69bcb62029a418
Abhilasha Ravichander
claimed_verified
2025-01-15T17:15:18.215Z
6349886c429608888c42319a
/avatars/f84b5fe8b76172878274754e3399d6ec.svg
Abhilasha Ravichander
user
lasha-nlp
6541c93cea69bcb62029a419
Dustin Schwenk
6541c93cea69bcb62029a41a
Alane Suhr
claimed_verified
2024-06-09T20:07:45.497Z
6611e6e1188ff298b0dd0b79
/avatars/3a495283955ec9e06e1829c7eb2cd9a4.svg
Alane Suhr
user
alsuhr
6541c93cea69bcb62029a41b
Pete Walsh
6541c93cea69bcb62029a41c
Dirk Groeneveld
claimed_verified
2024-01-04T09:34:18.870Z
60369745413a78f892e7339c
https://cdn-avatars.huggingface.co/v1/production/uploads/1636671879171-60369745413a78f892e7339c.png
Dirk Groeneveld
user
dirkgr
6541c93cea69bcb62029a41d
Luca Soldaini
claimed_verified
2024-03-25T09:44:54.820Z
5f04d8c45d08220171a0ad32
https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
Luca Soldaini
user
soldni
6541c93cea69bcb62029a41e
Sameer Singh
claimed_verified
2025-01-12T19:58:39.986Z
60b1206e6dfba500226fbf58
https://cdn-avatars.huggingface.co/v1/production/uploads/1622220905791-noauth.png
Sameer Singh
user
sameer
6541c93cea69bcb62029a41f
Hanna Hajishirzi
6541c93cea69bcb62029a420
Noah A. Smith
6541c93cea69bcb62029a421
Jesse Dodge
What's In My Big Data?
Large text corpora are the backbone of language models. However, we have a
limited understanding of the content of these corpora, including general
statistics, quality, social factors, and inclusion of evaluation data
(contamination). In this work, we propose What's In My Big Data? (WIMBD), a
platform and a set of sixteen analyses that allow us to reveal and compare the
contents of large text corpora. WIMBD builds on two basic capabilities -- count
and search -- at scale, which allows us to analyze more than 35 terabytes on a
standard compute node. We apply WIMBD to ten different corpora used to train
popular language models, including C4, The Pile, and RedPajama. Our analysis
uncovers several surprising and previously undocumented findings about these
corpora, including the high prevalence of duplicate, synthetic, and low-quality
content, personally identifiable information, toxic language, and benchmark
contamination. For instance, we find that about 50% of the documents in
RedPajama and LAION-2B-en are duplicates. In addition, several datasets used
for benchmarking models trained on such corpora are contaminated with respect
to important benchmarks, including the Winograd Schema Challenge and parts of
GLUE and SuperGLUE. We open-source WIMBD's code and artifacts to provide a
standard set of evaluations for new text-based corpora and to encourage more
analyses and transparency around them: github.com/allenai/wimbd.
WIMBD is a platform for analyzing large text corpora, revealing duplicates, synthetic content, and contamination in datasets used for language models.
WIMBD large text corpora language models text-based corpora