MathScale: Scaling Instruction Tuning for Mathematical Reasoning
Large language models (LLMs) have demonstrated remarkable capabilities in
problem-solving. However, their proficiency in solving mathematical problems
remains inadequate. We propose MathScale, a simple and scalable method to
create high-quality mathematical reasoning data using frontier LLMs (e.g., {\tt
GPT-3.5}). Inspired by the cognitive mechanism in human mathematical learning,
it first extracts topics and knowledge points from seed math questions and then
build a concept graph, which is subsequently used to generate new math
questions. MathScale exhibits effective scalability along the size axis of the
math dataset that we generate. As a result, we create a mathematical reasoning
dataset (MathScaleQA) containing two million math question-answer pairs. To
evaluate mathematical reasoning abilities of LLMs comprehensively, we construct
{\sc MwpBench}, a benchmark of Math Word Problems, which is a collection of ten
datasets (including GSM8K and MATH) covering K-12, college, and competition
level math problems. We apply MathScaleQA to fine-tune open-source LLMs (e.g.,
LLaMA-2 and Mistral), resulting in significantly improved capabilities in
mathematical reasoning. Evaluated on {\sc MwpBench}, MathScale-7B achieves
state-of-the-art performance across all datasets, surpassing its best peers of
equivalent size by 42.9\% in micro average accuracy and 43.7\% in macro average
accuracy, respectively.
65e7db6a7c10574cc3e7a8ef
Zhengyang Tang
admin_assigned
2024-03-06T09:59:41.936Z
64912976b95c3f0a1e6233cb
/avatars/c0615f8c6606073faffb419757d4e667.svg
Zhengyang Tang
user
tangzhy
65e7db6a7c10574cc3e7a8f0
Xingxing Zhang
admin_assigned
2024-03-06T09:59:48.538Z
64abbcff6cadc7aca584f71b
/avatars/fc6e85ad4a8befd133a37b411712c648.svg
Xingxing Zhang
user
THU-CHUNXIA
65e7db6a7c10574cc3e7a8f1
Benyou Wan
65e7db6a7c10574cc3e7a8f2
Furu Wei
admin_assigned
2024-03-06T10:00:04.844Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
MathScale: Scaling Instruction Tuning for Mathematical Reasoning
Large language models (LLMs) have demonstrated remarkable capabilities in
problem-solving. However, their proficiency in solving mathematical problems
remains inadequate. We propose MathScale, a simple and scalable method to
create high-quality mathematical reasoning data using frontier LLMs (e.g., {\tt
GPT-3.5}). Inspired by the cognitive mechanism in human mathematical learning,
it first extracts topics and knowledge points from seed math questions and then
build a concept graph, which is subsequently used to generate new math
questions. MathScale exhibits effective scalability along the size axis of the
math dataset that we generate. As a result, we create a mathematical reasoning
dataset (MathScaleQA) containing two million math question-answer pairs. To
evaluate mathematical reasoning abilities of LLMs comprehensively, we construct
{\sc MwpBench}, a benchmark of Math Word Problems, which is a collection of ten
datasets (including GSM8K and MATH) covering K-12, college, and competition
level math problems. We apply MathScaleQA to fine-tune open-source LLMs (e.g.,
LLaMA-2 and Mistral), resulting in significantly improved capabilities in
mathematical reasoning. Evaluated on {\sc MwpBench}, MathScale-7B achieves
state-of-the-art performance across all datasets, surpassing its best peers of
equivalent size by 42.9\% in micro average accuracy and 43.7\% in macro average
accuracy, respectively.
MathScale generates a large dataset of mathematical reasoning questions by building concept graphs from seed questions, enhancing open-source LLMs' performance on Math Word Problems.
large language models mathematical reasoning MathScale GPT-3.5 concept graph MathScaleQA MwpBench GSM8K MATH LLaMA-2 Mistral fine-tuning micro average accuracy macro average accuracy