Putnam-AXIOM: A Functional and Static Benchmark
Current mathematical reasoning benchmarks for large language models (LLMs)
are approaching saturation, with some achieving > 90% accuracy, and are
increasingly compromised by training-set contamination. We introduce
Putnam-AXIOM, a benchmark of 522 university-level competition problems drawn
from the prestigious William Lowell Putnam Mathematical Competition, and
Putnam-AXIOM Variation, an unseen companion set of 100 functional variants
generated by programmatically perturbing variables and constants. The variation
protocol produces an unlimited stream of equally difficult, unseen instances --
yielding a contamination-resilient test bed. On the Original set, OpenAI's
o1-preview -- the strongest evaluated model -- scores 41.9%, but its accuracy
drops by 19.6% (46.8% relative decrease) on the paired Variations. The
remaining eighteen models show the same downward trend, ten of them with
non-overlapping 95% confidence intervals. These gaps suggest memorization and
highlight the necessity of dynamic benchmarks. We complement "boxed" accuracy
with Teacher-Forced Accuracy (TFA), a lightweight metric that directly scores
reasoning traces and automates natural language proof evaluations. Putnam-AXIOM
therefore provides a rigorous, contamination-resilient evaluation framework for
assessing advanced mathematical reasoning of LLMs. Data and evaluation code are
publicly available at https://github.com/brando90/putnam-axiom.
689d1406b083e610d741e954
Aryan Gulati
689d1406b083e610d741e955
Brando Miranda
689d1406b083e610d741e956
Eric Chen
689d1406b083e610d741e957
Emily Xia
689d1406b083e610d741e958
Kai Fronsdal
689d1406b083e610d741e959
Bruno Dumont
689d1406b083e610d741e95a
Elyas Obbad
689d1406b083e610d741e95b
Sanmi Koyejo
Putnam-AXIOM: A Functional and Static Benchmark
Current mathematical reasoning benchmarks for large language models (LLMs)
are approaching saturation, with some achieving > 90% accuracy, and are
increasingly compromised by training-set contamination. We introduce
Putnam-AXIOM, a benchmark of 522 university-level competition problems drawn
from the prestigious William Lowell Putnam Mathematical Competition, and
Putnam-AXIOM Variation, an unseen companion set of 100 functional variants
generated by programmatically perturbing variables and constants. The variation
protocol produces an unlimited stream of equally difficult, unseen instances --
yielding a contamination-resilient test bed. On the Original set, OpenAI's
o1-preview -- the strongest evaluated model -- scores 41.9%, but its accuracy
drops by 19.6% (46.8% relative decrease) on the paired Variations. The
remaining eighteen models show the same downward trend, ten of them with
non-overlapping 95% confidence intervals. These gaps suggest memorization and
highlight the necessity of dynamic benchmarks. We complement "boxed" accuracy
with Teacher-Forced Accuracy (TFA), a lightweight metric that directly scores
reasoning traces and automates natural language proof evaluations. Putnam-AXIOM
therefore provides a rigorous, contamination-resilient evaluation framework for
assessing advanced mathematical reasoning of LLMs. Data and evaluation code are
publicly available at https://github.com/brando90/putnam-axiom.
Putnam-AXIOM and its variation set provide a contamination-resilient benchmark for evaluating advanced mathematical reasoning in large language models, revealing issues with memorization.
Putnam-AXIOM Putnam-AXIOM Variation Teacher-Forced Accuracy TFA mathematical reasoning large language models LLMs training-set contamination memorization dynamic benchmarks