Do PhD-level LLMs Truly Grasp Elementary Addition? Probing Rule Learning
  vs. Memorization in Large Language Models
Despite high benchmark scores, Large Language Models (LLMs) often fail simple
problem, raising a critical question: Do LLMs learn mathematical principles or
merely memorize patterns? Rather than designing increasingly complex benchmarks
like recent works, we investigate this using elementary two-integer addition
(0 to 2^{64}), probing two core properties: commutativity (A+B=B+A) and
compositional generalization (via isomorphic symbolic mappings, e.g., 7
rightarrow y). While state-of-the-art LLMs achieve 73.8-99.8\% accuracy on
numerical addition, performance collapses to leq7.5\% under symbolic
mapping, indicating failure to generalize learned rules. Non-monotonic
performance scaling with digit count and frequent commutativity violations
(over 1,700 cases of A+B neq B+A) further support this. Explicitly providing
addition rules degrades performance by 81.2\% on average, while
self-explanation maintains baseline accuracy, suggesting LLM arithmetic
processing is misaligned with human-defined principles. Our findings indicate
current LLMs rely on memory pattern over genuine rule learning, highlighting
architectural limitations and the need for new approaches to achieve true
mathematical reasoning.
67fcc9a980568c7ef6180dcb
Yang Yan
claimed_verified
2025-04-14T12:52:13.346Z
62cd5917299c0c2e0e435847
/avatars/b956b4feab86f6866c43cc87a44e25fc.svg
Yang Yan
user
kurileo
67fcc9a980568c7ef6180dcc
Yu Lu
67fcc9a980568c7ef6180dcd
Renjun Xu
67fcc9a980568c7ef6180dce
Zhenzhong Lan
Do PhD-level LLMs Truly Grasp Elementary Addition? Probing Rule Learning
  vs. Memorization in Large Language Models
Despite high benchmark scores, Large Language Models (LLMs) often fail simple
problem, raising a critical question: Do LLMs learn mathematical principles or
merely memorize patterns? Rather than designing increasingly complex benchmarks
like recent works, we investigate this using elementary two-integer addition
(0 to 2^{64}), probing two core properties: commutativity (A+B=B+A) and
compositional generalization (via isomorphic symbolic mappings, e.g., 7
rightarrow y). While state-of-the-art LLMs achieve 73.8-99.8\% accuracy on
numerical addition, performance collapses to leq7.5\% under symbolic
mapping, indicating failure to generalize learned rules. Non-monotonic
performance scaling with digit count and frequent commutativity violations
(over 1,700 cases of A+B neq B+A) further support this. Explicitly providing
addition rules degrades performance by 81.2\% on average, while
self-explanation maintains baseline accuracy, suggesting LLM arithmetic
processing is misaligned with human-defined principles. Our findings indicate
current LLMs rely on memory pattern over genuine rule learning, highlighting
architectural limitations and the need for new approaches to achieve true
mathematical reasoning.
LLMs achieve high accuracy in numerical addition but fail to generalize to symbolic mappings and exhibit poor performance in commutativity, suggesting they rely on memorization rather than understanding mathematical principles.
Large Language Models elementary two-integer addition commutativity compositional generalization isomorphic symbolic mappings self-explanation