MathCoder2: Better Math Reasoning from Continued Pretraining on
  Model-translated Mathematical Code
Code has been shown to be effective in enhancing the mathematical reasoning
abilities of large language models due to its precision and accuracy. Previous
works involving continued mathematical pretraining often include code that
utilizes math-related packages, which are primarily designed for fields such as
engineering, machine learning, signal processing, or module testing, rather
than being directly focused on mathematical reasoning. In this paper, we
introduce a novel method for generating mathematical code accompanied with
corresponding reasoning steps for continued pretraining. Our approach begins
with the construction of a high-quality mathematical continued pretraining
dataset by incorporating math-related web data, code using mathematical
packages, math textbooks, and synthetic data. Next, we construct reasoning
steps by extracting LaTeX expressions, the conditions needed for the
expressions, and the results of the expressions from the previously collected
dataset. Based on this extracted information, we generate corresponding code to
accurately capture the mathematical reasoning process. Appending the generated
code to each reasoning step results in data consisting of paired natural
language reasoning steps and their corresponding code. Combining this data with
the original dataset results in a 19.2B-token high-performing mathematical
pretraining corpus, which we name MathCode-Pile. Training several popular base
models with this corpus significantly improves their mathematical abilities,
leading to the creation of the MathCoder2 family of models. All of our data
processing and training code is open-sourced, ensuring full transparency and
easy reproducibility of the entire data collection and training pipeline. The
code is released at https://github.com/mathllm/MathCoder2 .
6708bbc0d1c640e5586d21fe
Zimu Lu
claimed_verified
2024-10-15T09:28:21.421Z
64b0bfef2f2f9c345b87e673
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/7O3zCr_IyGvtz0BUdKNKi.jpeg
Zimu Lu
user
luzimu
6708bbc0d1c640e5586d21ff
Aojun Zhou
claimed_verified
2024-10-11T07:43:33.832Z
637de1520d5bb06fbe5207a9
/avatars/1090851217270c5a858b13e013356d4f.svg
AJ.Zhou
user
AJZhou
6708bbc0d1c640e5586d2200
Ke Wang
claimed_verified
2024-10-11T07:43:31.392Z
64d592c28767727dffa1f002
/avatars/fe38bcac944a2742dc12c624e62d24ef.svg
WangKe
user
scikkk
6708bbc0d1c640e5586d2201
Houxing Ren
admin_assigned
2024-10-11T10:24:55.884Z
63e4b63dd6278c161be47f4b
https://cdn-avatars.huggingface.co/v1/production/uploads/63e4b63dd6278c161be47f4b/u0y1_o9WBqPJh8C8dimJ8.jpeg
Houxing Ren
user
Houxing
6708bbc0d1c640e5586d2202
Weikang Shi
admin_assigned
2024-10-11T10:25:02.066Z
6337e1d1107c4835a04c8607
/avatars/e85353b34c3da6efef84951934811fee.svg
Weikang Shi
user
shiwk20
6708bbc0d1c640e5586d2203
Junting Pan
admin_assigned
2024-10-11T10:25:08.002Z
639c2d36e62c431aa8d834a7
/avatars/a23d57dcb190711494a2a7e75e4cb9a8.svg
Junting Pan
user
juntingpan
6708bbc0d1c640e5586d2204
Mingjie Zhan
admin_assigned
2024-10-11T10:25:27.517Z
6301e46858f2b86bdfd49933
/avatars/d1c3fd94581f22a41506b24375a22bd3.svg
ZhanMingjie
user
Mingjie
6708bbc0d1c640e5586d2205
Hongsheng Li
admin_assigned
2024-10-11T10:25:33.604Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
MathCoder2: Better Math Reasoning from Continued Pretraining on
  Model-translated Mathematical Code
Code has been shown to be effective in enhancing the mathematical reasoning
abilities of large language models due to its precision and accuracy. Previous
works involving continued mathematical pretraining often include code that
utilizes math-related packages, which are primarily designed for fields such as
engineering, machine learning, signal processing, or module testing, rather
than being directly focused on mathematical reasoning. In this paper, we
introduce a novel method for generating mathematical code accompanied with
corresponding reasoning steps for continued pretraining. Our approach begins
with the construction of a high-quality mathematical continued pretraining
dataset by incorporating math-related web data, code using mathematical
packages, math textbooks, and synthetic data. Next, we construct reasoning
steps by extracting LaTeX expressions, the conditions needed for the
expressions, and the results of the expressions from the previously collected
dataset. Based on this extracted information, we generate corresponding code to
accurately capture the mathematical reasoning process. Appending the generated
code to each reasoning step results in data consisting of paired natural
language reasoning steps and their corresponding code. Combining this data with
the original dataset results in a 19.2B-token high-performing mathematical
pretraining corpus, which we name MathCode-Pile. Training several popular base
models with this corpus significantly improves their mathematical abilities,
leading to the creation of the MathCoder2 family of models. All of our data
processing and training code is open-sourced, ensuring full transparency and
easy reproducibility of the entire data collection and training pipeline. The
code is released at https://github.com/mathllm/MathCoder2 .
A novel method for generating mathematical code with reasoning steps enhances mathematical reasoning abilities in large language models using a comprehensive dataset named MathCode-Pile.
mathematical reasoning continued pretraining high-quality pretraining dataset LaTeX expressions synthetic data MathCode-Pile MathCoder2 family of models