Scaling Relationship on Learning Mathematical Reasoning with Large
  Language Models
Mathematical reasoning is a challenging task for large language models
(LLMs), while the scaling relationship of it with respect to LLM capacity is
under-explored. In this paper, we investigate how the pre-training loss,
supervised data amount, and augmented data amount influence the reasoning
performances of a supervised LLM. We find that pre-training loss is a better
indicator of the model's performance than the model's parameter count. We apply
supervised fine-tuning (SFT) with different amounts of supervised data and
empirically find a log-linear relation between data amount and model
performance, and we find better models improve less with enlarged supervised
datasets. To augment more data samples for improving model performances without
any human effort, we propose to apply Rejection sampling Fine-Tuning (RFT). RFT
uses supervised models to generate and collect correct reasoning paths as
augmented fine-tuning datasets. We find with augmented samples containing more
distinct reasoning paths, RFT improves mathematical reasoning performance more
for LLMs. We also find RFT brings more improvement for less performant LLMs.
Furthermore, we combine rejection samples from multiple models which push
LLaMA-7B to an accuracy of 49.3% and outperforms the supervised fine-tuning
(SFT) accuracy of 35.9% significantly.
64cc5655a257a3212c1588d4
Zheng Yuan
claimed_verified
2023-08-04T11:40:20.892Z
5f694b8b5e78cc6b0ed31bd2
/avatars/63f21d30918c34fff9ea592e1039b3f0.svg
Zheng Yuan
user
GanjinZero
64cc5655a257a3212c1588d5
Hongyi Yuan
claimed_verified
2023-08-04T15:05:10.506Z
64b77b602f5a966b9731b14a
/avatars/cdcf534bd451140dff47286095336535.svg
Hongyi Yuan
user
Yuan1997
64cc5655a257a3212c1588d6
Chengpeng Li
64cc5655a257a3212c1588d7
Guanting Dong
claimed_verified
2024-02-15T08:29:15.654Z
61cd4b833dd34ba1985e0753
https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
KABI
user
dongguanting
64cc5655a257a3212c1588d8
Chuanqi Tan
64cc5655a257a3212c1588d9
Chang Zhou
Scaling Relationship on Learning Mathematical Reasoning with Large
  Language Models
Mathematical reasoning is a challenging task for large language models
(LLMs), while the scaling relationship of it with respect to LLM capacity is
under-explored. In this paper, we investigate how the pre-training loss,
supervised data amount, and augmented data amount influence the reasoning
performances of a supervised LLM. We find that pre-training loss is a better
indicator of the model's performance than the model's parameter count. We apply
supervised fine-tuning (SFT) with different amounts of supervised data and
empirically find a log-linear relation between data amount and model
performance, and we find better models improve less with enlarged supervised
datasets. To augment more data samples for improving model performances without
any human effort, we propose to apply Rejection sampling Fine-Tuning (RFT). RFT
uses supervised models to generate and collect correct reasoning paths as
augmented fine-tuning datasets. We find with augmented samples containing more
distinct reasoning paths, RFT improves mathematical reasoning performance more
for LLMs. We also find RFT brings more improvement for less performant LLMs.
Furthermore, we combine rejection samples from multiple models which push
LLaMA-7B to an accuracy of 49.3% and outperforms the supervised fine-tuning
(SFT) accuracy of 35.9% significantly.
Rejection sampling fine-tuning enhances mathematical reasoning in large language models by generating auxiliary datasets of correct reasoning paths, leading to higher performance and better scalability compared to supervised fine-tuning.
pre-training loss supervised fine-tuning supervised data augmented data mathematical reasoning rejection sampling Fine-Tuning reasoning paths performance scalability LLaMA-7B