Language Models are Symbolic Learners in Arithmetic
Large Language Models (LLMs) are thought to struggle with arithmetic learning
due to the inherent differences between language modeling and numerical
computation, but concrete evidence has been lacking. This work responds to this
claim through a two-side experiment. We first investigate whether LLMs leverage
partial products during arithmetic learning. We find that although LLMs can
identify some partial products after learning, they fail to leverage them for
arithmetic tasks, conversely. We then explore how LLMs approach arithmetic
symbolically by breaking tasks into subgroups, hypothesizing that difficulties
arise from subgroup complexity and selection. Our results show that when
subgroup complexity is fixed, LLMs treat a collection of different arithmetic
operations similarly. By analyzing position-level accuracy across different
training sizes, we further observe that it follows a U-shaped pattern: LLMs
quickly learn the easiest patterns at the first and last positions, while
progressively learning the more difficult patterns in the middle positions.
This suggests that LLMs select subgroup following an easy-to-hard paradigm
during learning. Our work confirms that LLMs are pure symbolic learners in
arithmetic tasks and underscores the importance of understanding them deeply
through subgroup-level quantification.
671b23c017de97fe524d5828
Chunyuan Deng
claimed_verified
2024-10-30T09:54:46.917Z
634b9914dcf125e4da02498b
https://cdn-avatars.huggingface.co/v1/production/uploads/634b9914dcf125e4da02498b/crRgFroWq5U6XWtvlTXSZ.jpeg
Chunyuan Deng
user
CharlesDDDD
671b23c017de97fe524d5829
Zhiqi Li
671b23c017de97fe524d582a
Roy Xie
671b23c017de97fe524d582b
Ruidi Chang
671b23c017de97fe524d582c
Hanjie Chen
Language Models are Symbolic Learners in Arithmetic
Large Language Models (LLMs) are thought to struggle with arithmetic learning
due to the inherent differences between language modeling and numerical
computation, but concrete evidence has been lacking. This work responds to this
claim through a two-side experiment. We first investigate whether LLMs leverage
partial products during arithmetic learning. We find that although LLMs can
identify some partial products after learning, they fail to leverage them for
arithmetic tasks, conversely. We then explore how LLMs approach arithmetic
symbolically by breaking tasks into subgroups, hypothesizing that difficulties
arise from subgroup complexity and selection. Our results show that when
subgroup complexity is fixed, LLMs treat a collection of different arithmetic
operations similarly. By analyzing position-level accuracy across different
training sizes, we further observe that it follows a U-shaped pattern: LLMs
quickly learn the easiest patterns at the first and last positions, while
progressively learning the more difficult patterns in the middle positions.
This suggests that LLMs select subgroup following an easy-to-hard paradigm
during learning. Our work confirms that LLMs are pure symbolic learners in
arithmetic tasks and underscores the importance of understanding them deeply
through subgroup-level quantification.
LLMs struggle with arithmetic learning despite being able to identify partial products, treating different operations similarly when subgroup complexity is fixed, and learning patterns in an easy-to-hard manner.
Large Language Models LLMs arithmetic learning partial products symbolic learners subgroup complexity U-shaped pattern position-level accuracy