Generative AI for Math: Part I -- MathPile: A Billion-Token-Scale
  Pretraining Corpus for Math
High-quality, large-scale corpora are the cornerstone of building foundation
models. In this work, we introduce MathPile, a diverse and
high-quality math-centric corpus comprising about 9.5 billion tokens.
Throughout its creation, we adhered to the principle of ``less is
more'', firmly believing in the supremacy of data quality over quantity, even
in the pre-training phase. Our meticulous data collection and processing
efforts included a complex suite of preprocessing, prefiltering, language
identification, cleaning, filtering, and deduplication, ensuring the high
quality of our corpus. Furthermore, we performed data contamination detection
on downstream benchmark test sets to eliminate duplicates. We hope our
MathPile can help to enhance the mathematical reasoning abilities of
language models. We plan to open-source different versions of \mathpile with
the scripts used for processing, to facilitate future developments in this
field.
658e38c84a24a38778964cba
Zengzhi Wang
claimed_verified
2023-12-29T09:00:39.609Z
62cbeb2d72dfd24b86bdf977
https://cdn-avatars.huggingface.co/v1/production/uploads/62cbeb2d72dfd24b86bdf977/UcGYYSBNrCvPM5K9v-sro.png
Zengzhi Wang
user
SinclairWang
658e38c84a24a38778964cbb
Rui Xia
658e38c84a24a38778964cbc
Pengfei Liu
claimed_verified
2024-01-02T09:27:44.659Z
6144a0c4ff1146bbd84d9865
https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
Pengfei Liu
user
Pengfei
Generative AI for Math: Part I -- MathPile: A Billion-Token-Scale
  Pretraining Corpus for Math
High-quality, large-scale corpora are the cornerstone of building foundation
models. In this work, we introduce MathPile, a diverse and
high-quality math-centric corpus comprising about 9.5 billion tokens.
Throughout its creation, we adhered to the principle of ``less is
more'', firmly believing in the supremacy of data quality over quantity, even
in the pre-training phase. Our meticulous data collection and processing
efforts included a complex suite of preprocessing, prefiltering, language
identification, cleaning, filtering, and deduplication, ensuring the high
quality of our corpus. Furthermore, we performed data contamination detection
on downstream benchmark test sets to eliminate duplicates. We hope our
MathPile can help to enhance the mathematical reasoning abilities of
language models. We plan to open-source different versions of \mathpile with
the scripts used for processing, to facilitate future developments in this
field.
A new large-scale, high-quality math-centric corpus named MathPile has been introduced to enhance mathematical reasoning abilities of language models.