Large Language Models and Mathematical Reasoning Failures
This paper investigates the mathematical reasoning capabilities of large
language models (LLMs) using 50 newly constructed high-school-level word
problems. Unlike prior studies that focus solely on answer correctness, we
rigorously analyze both final answers and solution steps to identify reasoning
failures. Evaluating eight state-of-the-art models - including Mixtral, Llama,
Gemini, GPT-4o, and OpenAI's o1 variants - we find that while newer models
(e.g., o3-mini, deepseek-r1) achieve higher accuracy, all models exhibit errors
in spatial reasoning, strategic planning, and arithmetic, sometimes producing
correct answers through flawed logic. Common failure modes include unwarranted
assumptions, over-reliance on numerical patterns, and difficulty translating
physical intuition into mathematical steps. Manual analysis reveals that models
struggle with problems requiring multi-step deduction or real-world knowledge,
despite possessing broad mathematical knowledge. Our results underscore the
importance of evaluating reasoning processes, not just answers, and caution
against overestimating LLMs' problem-solving proficiency. The study highlights
persistent gaps in LLMs' generalization abilities, emphasizing the need for
targeted improvements in structured reasoning and constraint handling.
67b435c29e5685b308a8edac
Johan Boye
extracted_pending
2025-02-18T07:24:50.956Z
65bcbc01d6d0ffbceb8b2e6e
/avatars/73edb2d6b7b11208439ac88b365079e8.svg
Johan Boye
user
jboye
67b435c29e5685b308a8edad
Birger Moell
claimed_verified
2025-02-18T09:30:49.328Z
6033e34a9aa44495c80dd043
https://cdn-avatars.huggingface.co/v1/production/uploads/1614079701740-6033e34a9aa44495c80dd043.jpeg
Birger Moell
user
birgermoell
Large Language Models and Mathematical Reasoning Failures
This paper investigates the mathematical reasoning capabilities of large
language models (LLMs) using 50 newly constructed high-school-level word
problems. Unlike prior studies that focus solely on answer correctness, we
rigorously analyze both final answers and solution steps to identify reasoning
failures. Evaluating eight state-of-the-art models - including Mixtral, Llama,
Gemini, GPT-4o, and OpenAI's o1 variants - we find that while newer models
(e.g., o3-mini, deepseek-r1) achieve higher accuracy, all models exhibit errors
in spatial reasoning, strategic planning, and arithmetic, sometimes producing
correct answers through flawed logic. Common failure modes include unwarranted
assumptions, over-reliance on numerical patterns, and difficulty translating
physical intuition into mathematical steps. Manual analysis reveals that models
struggle with problems requiring multi-step deduction or real-world knowledge,
despite possessing broad mathematical knowledge. Our results underscore the
importance of evaluating reasoning processes, not just answers, and caution
against overestimating LLMs' problem-solving proficiency. The study highlights
persistent gaps in LLMs' generalization abilities, emphasizing the need for
targeted improvements in structured reasoning and constraint handling.
mathematical reasoning capabilities large language models (LLMs) word problems solution steps reasoning failures adaptively scheduling Mixtral Llama Gemini GPT-4 OpenAI accuracy spatial reasoning strategic planning arithmetic flawed logic unwarranted assumptions over-reliance on numerical patterns multi-step deduction real-world knowledge multimodal analysis generalization abilities structured reasoning constraint handling