Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to
  the Edge of Generalization
We study whether transformers can learn to implicitly reason over parametric
knowledge, a skill that even the most capable language models struggle with.
Focusing on two representative reasoning types, composition and comparison, we
consistently find that transformers can learn implicit reasoning, but only
through grokking, i.e., extended training far beyond overfitting. The levels of
generalization also vary across reasoning types: when faced with
out-of-distribution examples, transformers fail to systematically generalize
for composition but succeed for comparison. We delve into the model's internals
throughout training, conducting analytical experiments that reveal: 1) the
mechanism behind grokking, such as the formation of the generalizing circuit
and its relation to the relative efficiency of generalizing and memorizing
circuits, and 2) the connection between systematicity and the configuration of
the generalizing circuit. Our findings guide data and training setup to better
induce implicit reasoning and suggest potential improvements to the transformer
architecture, such as encouraging cross-layer knowledge sharing. Furthermore,
we demonstrate that for a challenging reasoning task with a large search space,
GPT-4-Turbo and Gemini-1.5-Pro based on non-parametric memory fail badly
regardless of prompting styles or retrieval augmentation, while a fully grokked
transformer can achieve near-perfect accuracy, showcasing the power of
parametric memory for complex reasoning.
6653e9e7f8137bb650b40fab
Boshi Wang
admin_assigned
2024-05-27T08:09:24.353Z
62e5e742972081fe55828530
/avatars/1cc9a4d5fbb634879a31f45bb7a5ec41.svg
Boshi Wang
user
boshiwang
6653e9e7f8137bb650b40fac
Xiang Yue
claimed_verified
2024-05-27T07:18:12.062Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
6653e9e7f8137bb650b40fad
Yu Su
claimed_verified
2024-10-14T19:21:59.248Z
6477a323dbc2a416f8b852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
Yu Su
user
ysu-nlp
6653e9e7f8137bb650b40fae
Huan Sun
admin_assigned
2024-05-27T08:09:45.562Z
6127b327699a5d2cde0b44dc
https://cdn-avatars.huggingface.co/v1/production/uploads/1629991702069-noauth.png
Huan Sun
user
huansun
Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to
  the Edge of Generalization
We study whether transformers can learn to implicitly reason over parametric
knowledge, a skill that even the most capable language models struggle with.
Focusing on two representative reasoning types, composition and comparison, we
consistently find that transformers can learn implicit reasoning, but only
through grokking, i.e., extended training far beyond overfitting. The levels of
generalization also vary across reasoning types: when faced with
out-of-distribution examples, transformers fail to systematically generalize
for composition but succeed for comparison. We delve into the model's internals
throughout training, conducting analytical experiments that reveal: 1) the
mechanism behind grokking, such as the formation of the generalizing circuit
and its relation to the relative efficiency of generalizing and memorizing
circuits, and 2) the connection between systematicity and the configuration of
the generalizing circuit. Our findings guide data and training setup to better
induce implicit reasoning and suggest potential improvements to the transformer
architecture, such as encouraging cross-layer knowledge sharing. Furthermore,
we demonstrate that for a challenging reasoning task with a large search space,
GPT-4-Turbo and Gemini-1.5-Pro based on non-parametric memory fail badly
regardless of prompting styles or retrieval augmentation, while a fully grokked
transformer can achieve near-perfect accuracy, showcasing the power of
parametric memory for complex reasoning.
Transformers can learn implicit reasoning through extensive training, with varying degrees of generalization for different reasoning types, and parametric memory proves more effective for complex reasoning tasks compared to non-parametric memory.
transformers implicit reasoning grokking overfitting composition comparison generalization generalizing circuit memorizing circuits systematicity cross-layer knowledge sharing parametric memory non-parametric memory GPT-4-Turbo Gemini-1.5-Pro