GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in
  Large Language Models
Recent advancements in Large Language Models (LLMs) have sparked interest in
their formal reasoning capabilities, particularly in mathematics. The GSM8K
benchmark is widely used to assess the mathematical reasoning of models on
grade-school-level questions. While the performance of LLMs on GSM8K has
significantly improved in recent years, it remains unclear whether their
mathematical reasoning capabilities have genuinely advanced, raising questions
about the reliability of the reported metrics. To address these concerns, we
conduct a large-scale study on several SOTA open and closed models. To overcome
the limitations of existing evaluations, we introduce GSM-Symbolic, an improved
benchmark created from symbolic templates that allow for the generation of a
diverse set of questions. GSM-Symbolic enables more controllable evaluations,
providing key insights and more reliable metrics for measuring the reasoning
capabilities of models.Our findings reveal that LLMs exhibit noticeable
variance when responding to different instantiations of the same question.
Specifically, the performance of all models declines when only the numerical
values in the question are altered in the GSM-Symbolic benchmark. Furthermore,
we investigate the fragility of mathematical reasoning in these models and show
that their performance significantly deteriorates as the number of clauses in a
question increases. We hypothesize that this decline is because current LLMs
cannot perform genuine logical reasoning; they replicate reasoning steps from
their training data. Adding a single clause that seems relevant to the question
causes significant performance drops (up to 65%) across all state-of-the-art
models, even though the clause doesn't contribute to the reasoning chain needed
for the final answer. Overall, our work offers a more nuanced understanding of
LLMs' capabilities and limitations in mathematical reasoning.
67052d96d2d7aab4e5a990a9
Iman Mirzadeh
67052d96d2d7aab4e5a990aa
Keivan Alizadeh
67052d96d2d7aab4e5a990ab
Hooman Shahrokhi
67052d96d2d7aab4e5a990ac
Oncel Tuzel
67052d96d2d7aab4e5a990ad
Samy Bengio
67052d96d2d7aab4e5a990ae
Mehrdad Farajtabar
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in
  Large Language Models
Recent advancements in Large Language Models (LLMs) have sparked interest in
their formal reasoning capabilities, particularly in mathematics. The GSM8K
benchmark is widely used to assess the mathematical reasoning of models on
grade-school-level questions. While the performance of LLMs on GSM8K has
significantly improved in recent years, it remains unclear whether their
mathematical reasoning capabilities have genuinely advanced, raising questions
about the reliability of the reported metrics. To address these concerns, we
conduct a large-scale study on several SOTA open and closed models. To overcome
the limitations of existing evaluations, we introduce GSM-Symbolic, an improved
benchmark created from symbolic templates that allow for the generation of a
diverse set of questions. GSM-Symbolic enables more controllable evaluations,
providing key insights and more reliable metrics for measuring the reasoning
capabilities of models.Our findings reveal that LLMs exhibit noticeable
variance when responding to different instantiations of the same question.
Specifically, the performance of all models declines when only the numerical
values in the question are altered in the GSM-Symbolic benchmark. Furthermore,
we investigate the fragility of mathematical reasoning in these models and show
that their performance significantly deteriorates as the number of clauses in a
question increases. We hypothesize that this decline is because current LLMs
cannot perform genuine logical reasoning; they replicate reasoning steps from
their training data. Adding a single clause that seems relevant to the question
causes significant performance drops (up to 65%) across all state-of-the-art
models, even though the clause doesn't contribute to the reasoning chain needed
for the final answer. Overall, our work offers a more nuanced understanding of
LLMs' capabilities and limitations in mathematical reasoning.
LSSMs exhibit variability and fragility in mathematical reasoning, as evidenced by performance declines on GSM-Symbolic benchmark questions with altered numerical values and additional clauses.
Large Language Models mathematical reasoning GSM8K benchmark GSM-Symbolic symbolic templates logical reasoning