MegaMath: Pushing the Limits of Open Math Corpora
Mathematical reasoning is a cornerstone of human intelligence and a key
benchmark for advanced capabilities in large language models (LLMs). However,
the research community still lacks an open, large-scale, high-quality corpus
tailored to the demands of math-centric LLM pre-training. We present MegaMath,
an open dataset curated from diverse, math-focused sources through following
practices: (1) Revisiting web data: We re-extracted mathematical documents from
Common Crawl with math-oriented HTML optimizations, fasttext-based filtering
and deduplication, all for acquiring higher-quality data on the Internet. (2)
Recalling Math-related code data: We identified high quality math-related code
from large code training corpus, Stack-V2, further enhancing data diversity.
(3) Exploring Synthetic data: We synthesized QA-style text, math-related code,
and interleaved text-code blocks from web data or code data. By integrating
these strategies and validating their effectiveness through extensive
ablations, MegaMath delivers 371B tokens with the largest quantity and top
quality among existing open math pre-training datasets.
67efe6e839d83c66f96ef051
Fan Zhou
claimed_verified
2025-04-08T06:54:00.099Z
628f6e5ab90dde28ef57d293
https://cdn-avatars.huggingface.co/v1/production/uploads/628f6e5ab90dde28ef57d293/AxNzR2nvrND6Rf3RPkYMk.jpeg
Fan Zhou
user
koalazf99
67efe6e839d83c66f96ef052
Zengzhi Wang
claimed_verified
2025-06-27T08:58:23.784Z
62cbeb2d72dfd24b86bdf977
https://cdn-avatars.huggingface.co/v1/production/uploads/62cbeb2d72dfd24b86bdf977/UcGYYSBNrCvPM5K9v-sro.png
Zengzhi Wang
user
SinclairWang
67efe6e839d83c66f96ef053
Nikhil Ranjan
67efe6e839d83c66f96ef054
Zhoujun Cheng
67efe6e839d83c66f96ef055
Liping Tang
67efe6e839d83c66f96ef056
Guowei He
67efe6e839d83c66f96ef057
Zhengzhong Liu
67efe6e839d83c66f96ef058
Eric P. Xing
MegaMath: Pushing the Limits of Open Math Corpora
Mathematical reasoning is a cornerstone of human intelligence and a key
benchmark for advanced capabilities in large language models (LLMs). However,
the research community still lacks an open, large-scale, high-quality corpus
tailored to the demands of math-centric LLM pre-training. We present MegaMath,
an open dataset curated from diverse, math-focused sources through following
practices: (1) Revisiting web data: We re-extracted mathematical documents from
Common Crawl with math-oriented HTML optimizations, fasttext-based filtering
and deduplication, all for acquiring higher-quality data on the Internet. (2)
Recalling Math-related code data: We identified high quality math-related code
from large code training corpus, Stack-V2, further enhancing data diversity.
(3) Exploring Synthetic data: We synthesized QA-style text, math-related code,
and interleaved text-code blocks from web data or code data. By integrating
these strategies and validating their effectiveness through extensive
ablations, MegaMath delivers 371B tokens with the largest quantity and top
quality among existing open math pre-training datasets.
MegaMath is an open dataset designed for math-centric LLM pre-training, combining high-quality web data, math-related code, and synthetic content to enhance diversity and quality.
LLMs MegaMath Common Crawl HTML optimizations fastText deduplication Stack-V2 QA-style text interleaved text-code blocks math pre-training datasets