A Careful Examination of Large Language Model Performance on Grade
  School Arithmetic
Large language models (LLMs) have achieved impressive success on many
benchmarks for mathematical reasoning. However, there is growing concern that
some of this performance actually reflects dataset contamination, where data
closely resembling benchmark questions leaks into the training data, instead of
true reasoning ability. To investigate this claim rigorously, we commission
Grade School Math 1000 (GSM1k). GSM1k is designed to mirror the style and
complexity of the established GSM8k benchmark, the gold standard for measuring
elementary mathematical reasoning. We ensure that the two benchmarks are
comparable across important metrics such as human solve rates, number of steps
in solution, answer magnitude, and more. When evaluating leading open- and
closed-source LLMs on GSM1k, we observe accuracy drops of up to 13%, with
several families of models (e.g., Phi and Mistral) showing evidence of
systematic overfitting across almost all model sizes. At the same time, many
models, especially those on the frontier, (e.g., Gemini/GPT/Claude) show
minimal signs of overfitting. Further analysis suggests a positive relationship
(Spearman's r^2=0.32) between a model's probability of generating an example
from GSM8k and its performance gap between GSM8k and GSM1k, suggesting that
many models may have partially memorized GSM8k.
663300c27dfc25dfbf8f0250
Hugh Zhang
extracted_confirmed
2024-05-02T07:48:47.067Z
65d6a5f94c28026a003581b4
/avatars/c24c314eafbf6a885c586258ae87a47f.svg
Hugh Zhang
user
hugh-scale
663300c27dfc25dfbf8f0251
Jeff Da
663300c27dfc25dfbf8f0252
Dean Lee
663300c27dfc25dfbf8f0253
Vaughn Robinson
admin_assigned
2024-05-02T07:23:24.254Z
64e259ab93103b4537bfc815
/avatars/664c31c467fbae50d730b0c9df891dd2.svg
Vaughn Robinson
user
dumela123
663300c27dfc25dfbf8f0254
Catherine Wu
admin_assigned
2024-05-02T07:23:30.378Z
632f99c62a6ef6fb4ad53949
/avatars/ad25b15d599fa21c2a75152ceb01f9ca.svg
Catherine Wu
user
catwu
663300c27dfc25dfbf8f0255
Will Song
663300c27dfc25dfbf8f0256
Tiffany Zhao
663300c27dfc25dfbf8f0257
Pranav Raja
admin_assigned
2024-05-02T07:23:59.291Z
63f291970a16587ea96c4735
/avatars/6d3f2029a7ef1f0f8b47c1d9829d9aa0.svg
Pranav Raja
user
pranavrajascale
663300c27dfc25dfbf8f0258
Dylan Slack
admin_assigned
2024-05-02T07:24:20.238Z
62ec892ff105ff23a9bd4548
https://cdn-avatars.huggingface.co/v1/production/uploads/1659668761477-noauth.png
Dylan Slack
user
dslack
663300c27dfc25dfbf8f0259
Qin Lyu
admin_assigned
2024-05-02T07:24:27.051Z
64b6dd32aaebda400ce31b05
/avatars/4ff1783e873261c3874f8752e81e5a77.svg
Qin Lyu
user
lyuqin
663300c27dfc25dfbf8f025a
Sean Hendryx
claimed_verified
2024-10-17T11:29:56.193Z
67106d78607722fac0b844a7
/avatars/459b1a39541f1dd871aad66e653aed48.svg
Sean Hendryx
user
hendryx-scale
663300c27dfc25dfbf8f025b
Russell Kaplan
admin_assigned
2024-05-02T07:24:36.607Z
6313cc48c093ff968e0d9c5c
/avatars/7f0aa8ee6274d0a7790ef0b3298ed19b.svg
Russell Kaplan
user
rkaplan
663300c27dfc25dfbf8f025c
Michele
663300c27dfc25dfbf8f025d
Lunati
663300c27dfc25dfbf8f025e
Summer Yue
A Careful Examination of Large Language Model Performance on Grade
  School Arithmetic
Large language models (LLMs) have achieved impressive success on many
benchmarks for mathematical reasoning. However, there is growing concern that
some of this performance actually reflects dataset contamination, where data
closely resembling benchmark questions leaks into the training data, instead of
true reasoning ability. To investigate this claim rigorously, we commission
Grade School Math 1000 (GSM1k). GSM1k is designed to mirror the style and
complexity of the established GSM8k benchmark, the gold standard for measuring
elementary mathematical reasoning. We ensure that the two benchmarks are
comparable across important metrics such as human solve rates, number of steps
in solution, answer magnitude, and more. When evaluating leading open- and
closed-source LLMs on GSM1k, we observe accuracy drops of up to 13%, with
several families of models (e.g., Phi and Mistral) showing evidence of
systematic overfitting across almost all model sizes. At the same time, many
models, especially those on the frontier, (e.g., Gemini/GPT/Claude) show
minimal signs of overfitting. Further analysis suggests a positive relationship
(Spearman's r^2=0.32) between a model's probability of generating an example
from GSM8k and its performance gap between GSM8k and GSM1k, suggesting that
many models may have partially memorized GSM8k.
Evaluation of large language models on a new math benchmark, GSM1k, reveals that many models exhibit signs of overfitting to the existing GSM8k benchmark, leading to performance drops on GSM1k.
large language models mathematical reasoning dataset contamination Grade School Math 1000 GSM1k GSM8k human solve rates solution steps answer magnitude system overfitting Spearman's r^2