Large Language Models and Mathematical Reasoning Failures
This paper investigates the mathematical reasoning capabilities of large
language models (LLMs) using 50 newly constructed high-school-level word
problems. Unlike prior studies that focus solely on answer correctness, we
rigorously analyze both final answers and solution steps to identify reasoning
failures. Evaluating eight state-of-the-art models - including Mixtral, Llama,
Gemini, GPT-4o, and OpenAI's o1 variants - we find that while newer models
(e.g., o3-mini, deepseek-r1) achieve higher accuracy, all models exhibit errors
in spatial reasoning, strategic planning, and arithmetic, sometimes producing
correct answers through flawed logic. Common failure modes include unwarranted
assumptions, over-reliance on numerical patterns, and difficulty translating
physical intuition into mathematical steps. Manual analysis reveals that models
struggle with problems requiring multi-step deduction or real-world knowledge,
despite possessing broad mathematical knowledge. Our results underscore the
importance of evaluating reasoning processes, not just answers, and caution
against overestimating LLMs' problem-solving proficiency. The study highlights
persistent gaps in LLMs' generalization abilities, emphasizing the need for
targeted improvements in structured reasoning and constraint handling.
67b435c29e5685b308a8edac
Johan Boye
extracted_pending
2025-02-18T07:24:50.956Z
65bcbc01d6d0ffbceb8b2e6e
/avatars/73edb2d6b7b11208439ac88b365079e8.svg
Johan Boye
user
jboye
67b435c29e5685b308a8edad
Birger Moell
claimed_verified
2025-02-18T09:30:49.328Z
6033e34a9aa44495c80dd043
https://cdn-avatars.huggingface.co/v1/production/uploads/1614079701740-6033e34a9aa44495c80dd043.jpeg
Birger Moell
user
birgermoell
Large Language Models and Mathematical Reasoning Failures
This paper investigates the mathematical reasoning capabilities of large
language models (LLMs) using 50 newly constructed high-school-level word
problems. Unlike prior studies that focus solely on answer correctness, we
rigorously analyze both final answers and solution steps to identify reasoning
failures. Evaluating eight state-of-the-art models - including Mixtral, Llama,
Gemini, GPT-4o, and OpenAI's o1 variants - we find that while newer models
(e.g., o3-mini, deepseek-r1) achieve higher accuracy, all models exhibit errors
in spatial reasoning, strategic planning, and arithmetic, sometimes producing
correct answers through flawed logic. Common failure modes include unwarranted
assumptions, over-reliance on numerical patterns, and difficulty translating
physical intuition into mathematical steps. Manual analysis reveals that models
struggle with problems requiring multi-step deduction or real-world knowledge,
despite possessing broad mathematical knowledge. Our results underscore the
importance of evaluating reasoning processes, not just answers, and caution
against overestimating LLMs' problem-solving proficiency. The study highlights
persistent gaps in LLMs' generalization abilities, emphasizing the need for
targeted improvements in structured reasoning and constraint handling.
Evaluation of large language models on high-school-level math problems shows that while newer models improve accuracy, they still fail in spatial reasoning, strategic planning, arithmetic, and multi-step deduction.
mathematical reasoning large language models word problems Mixtral Llama Gemini GPT-4 spatial reasoning strategic planning arithmetic solution steps reasoning failures manual analysis structured reasoning constraint handling