Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large
  Language Models -- The Story Goes On
In this paper, we investigate the underlying factors that potentially enhance
the mathematical reasoning capabilities of large language models (LLMs). We
argue that the data scaling law for math reasoning capabilities in modern LLMs
is far from being saturated, highlighting how the model's quality improves with
increases in data quantity. To support this claim, we introduce the
Skywork-Math model series, supervised fine-tuned (SFT) on common 7B LLMs using
our proposed 2.5M-instance Skywork-MathQA dataset. Skywork-Math 7B has achieved
impressive accuracies of 51.2% on the competition-level MATH benchmark and
83.9% on the GSM8K benchmark using only SFT data, outperforming an early
version of GPT-4 on MATH. The superior performance of Skywork-Math models
contributes to our novel two-stage data synthesis and model SFT pipelines,
which include three different augmentation methods and a diverse seed problem
set, ensuring both the quantity and quality of Skywork-MathQA dataset across
varying difficulty levels. Most importantly, we provide several practical
takeaways to enhance math reasoning abilities in LLMs for both research and
industry applications.
669096575cb4c07caa232efd
Liang Zeng
admin_assigned
2024-07-12T07:44:28.572Z
6621efe1a6eec3ad03e38759
/avatars/c35acce69f244ec0833dffd53eedf6a3.svg
Liang Zeng
user
zengliangcs
669096575cb4c07caa232efe
Liangjun Zhong
669096575cb4c07caa232eff
Liang Zhao
claimed_verified
2024-07-12T07:40:06.238Z
64996ce2ed09df36eaf236ea
https://cdn-avatars.huggingface.co/v1/production/uploads/64996ce2ed09df36eaf236ea/wIr1j2zkAezXfpsczTjq_.jpeg
zhaoliang
user
zhao1iang
669096575cb4c07caa232f00
Tianwen Wei
admin_assigned
2024-07-12T07:55:17.509Z
63d8773a635499f21644601b
/avatars/07efea204d172e2844dbacc418131da2.svg
Wei Tianwen
user
weitianwen
669096575cb4c07caa232f01
Liu Yang
669096575cb4c07caa232f02
Jujie He
admin_assigned
2024-07-12T07:55:46.201Z
64a676f223caaba1d2eb5d02
/avatars/c51a938e3b0e64f1aa38e6c63828e921.svg
Jujie He
user
leafzs
669096575cb4c07caa232f03
Cheng Cheng
669096575cb4c07caa232f04
Rui Hu
669096575cb4c07caa232f05
Yang Liu
669096575cb4c07caa232f06
Shuicheng Yan
669096575cb4c07caa232f07
Han Fang
669096575cb4c07caa232f08
Yahui Zhou
Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large
  Language Models -- The Story Goes On
In this paper, we investigate the underlying factors that potentially enhance
the mathematical reasoning capabilities of large language models (LLMs). We
argue that the data scaling law for math reasoning capabilities in modern LLMs
is far from being saturated, highlighting how the model's quality improves with
increases in data quantity. To support this claim, we introduce the
Skywork-Math model series, supervised fine-tuned (SFT) on common 7B LLMs using
our proposed 2.5M-instance Skywork-MathQA dataset. Skywork-Math 7B has achieved
impressive accuracies of 51.2% on the competition-level MATH benchmark and
83.9% on the GSM8K benchmark using only SFT data, outperforming an early
version of GPT-4 on MATH. The superior performance of Skywork-Math models
contributes to our novel two-stage data synthesis and model SFT pipelines,
which include three different augmentation methods and a diverse seed problem
set, ensuring both the quantity and quality of Skywork-MathQA dataset across
varying difficulty levels. Most importantly, we provide several practical
takeaways to enhance math reasoning abilities in LLMs for both research and
industry applications.
Skywork-Math model series achieves high performance in math reasoning by fine-tuning on a large and diverse dataset, demonstrating that data quantity significantly enhances LLM mathematical capabilities.
large language models data scaling law Skywork-Math model series supervised fine-tuned Skywork-MathQA dataset MATH benchmark GSM8K benchmark GPT-4 two-stage data synthesis model SFT pipelines data augmentation seed problem set