TinyGSM: achieving >80% on GSM8k with small language models
Small-scale models offer various computational advantages, and yet to which
extent size is critical for problem-solving abilities remains an open question.
Specifically for solving grade school math, the smallest model size so far
required to break the 80\% barrier on the GSM8K benchmark remains to be 34B.
Our work studies how high-quality datasets may be the key for small language
models to acquire mathematical reasoning. We introduce TinyGSM, a
synthetic dataset of 12.3M grade school math problems paired with Python
solutions, generated fully by GPT-3.5. After finetuning on TinyGSM, we
find that a duo of a 1.3B generation model and a 1.3B verifier model can
achieve 81.5\% accuracy, outperforming existing models that are orders of
magnitude larger. This also rivals the performance of the GPT-3.5 ``teacher''
model (77.4\%), from which our model's training data is generated. Our approach
is simple and has two key components: 1) the high-quality dataset
TinyGSM, 2) the use of a verifier, which selects the final outputs
from multiple candidate generations.
657bcee9696ec3dda930e4c4
Bingbin Liu
claimed_verified
2024-07-11T08:28:02.121Z
63d2e5ef2727d7888cb940b8
https://cdn-avatars.huggingface.co/v1/production/uploads/1674794281493-63d2e5ef2727d7888cb940b8.jpeg
Bingbin
user
ClaraBing
657bcee9696ec3dda930e4c5
Sebastien Bubeck
admin_assigned
2023-12-15T11:12:44.670Z
64557408c9c0dcc8c24a7a92
/avatars/911ba8fc3c224578d5946dddd94bf4c0.svg
Sebastien Bubeck
user
sebubeck
657bcee9696ec3dda930e4c6
Ronen Eldan
admin_assigned
2023-12-15T11:13:01.769Z
633723a80267ebcf0264c06b
/avatars/22bb971597e9f3abfa343280a9d0f65f.svg
Ronen Eldan
user
roneneldan
657bcee9696ec3dda930e4c7
Janardhan Kulkarni
657bcee9696ec3dda930e4c8
Yuanzhi Li
admin_assigned
2023-12-15T11:13:27.761Z
641d5a313f04f9bf2da17e8c
/avatars/4035f73c69ddb045573ad96db4f13f04.svg
Yuanzhi Li
user
Uushizhu1234
657bcee9696ec3dda930e4c9
Anh Nguyen
claimed_verified
2024-09-11T07:49:08.871Z
61ee4dc39539bc0a3bbc8a68
/avatars/723f89e9c7afecdbd2854544bd7eb083.svg
Anh Tuan Nguyen
user
tuananh7198
657bcee9696ec3dda930e4ca
Rachel Ward
657bcee9696ec3dda930e4cb
Yi Zhang
TinyGSM: achieving >80% on GSM8k with small language models
Small-scale models offer various computational advantages, and yet to which
extent size is critical for problem-solving abilities remains an open question.
Specifically for solving grade school math, the smallest model size so far
required to break the 80\% barrier on the GSM8K benchmark remains to be 34B.
Our work studies how high-quality datasets may be the key for small language
models to acquire mathematical reasoning. We introduce TinyGSM, a
synthetic dataset of 12.3M grade school math problems paired with Python
solutions, generated fully by GPT-3.5. After finetuning on TinyGSM, we
find that a duo of a 1.3B generation model and a 1.3B verifier model can
achieve 81.5\% accuracy, outperforming existing models that are orders of
magnitude larger. This also rivals the performance of the GPT-3.5 ``teacher''
model (77.4\%), from which our model's training data is generated. Our approach
is simple and has two key components: 1) the high-quality dataset
TinyGSM, 2) the use of a verifier, which selects the final outputs
from multiple candidate generations.
A synthetic dataset of math problems paired with solutions enables small language models to achieve high accuracy in mathematical reasoning, outperforming larger models.
TinyGSM GPT-3.5 mathematical reasoning verifier candidate generations