Your Transformer is Secretly Linear
This paper reveals a novel linear characteristic exclusive to transformer
decoders, including models such as GPT, LLaMA, OPT, BLOOM and others. We
analyze embedding transformations between sequential layers, uncovering a
near-perfect linear relationship (Procrustes similarity score of 0.99).
However, linearity decreases when the residual component is removed due to a
consistently low output norm of the transformer layer. Our experiments show
that removing or linearly approximating some of the most linear blocks of
transformers does not affect significantly the loss or model performance.
Moreover, in our pretraining experiments on smaller models we introduce a
cosine-similarity-based regularization, aimed at reducing layer linearity. This
regularization improves performance metrics on benchmarks like Tiny Stories and
SuperGLUE and as well successfully decreases the linearity of the models. This
study challenges the existing understanding of transformer architectures,
suggesting that their operation may be more linear than previously assumed.
664d6ab7d85a68a939cc73f4
Anton Razzhigaev
admin_assigned
2024-05-22T07:34:57.554Z
6172aaeec8e66e2aa84c06b9
https://cdn-avatars.huggingface.co/v1/production/uploads/6172aaeec8e66e2aa84c06b9/ZdRZSp3P1SU6CIDbvQwkv.jpeg
Anton Razzhigaev
user
razzant
664d6ab7d85a68a939cc73f5
Matvey Mikhalchuk
admin_assigned
2024-05-22T07:35:15.823Z
64ee45a944f4b3b1bccc02d1
https://cdn-avatars.huggingface.co/v1/production/uploads/64ee45a944f4b3b1bccc02d1/SoidO9HQ4mftzbUPtuBBf.png
Matvey Mikhalchuk
user
matveymih
664d6ab7d85a68a939cc73f6
Elizaveta Goncharova
admin_assigned
2024-05-22T07:35:22.290Z
6310ff34bc152fa3e810c186
/avatars/bfd63bcd81548283f5e496e3693bf143.svg
Elizaveta Goncharova
user
Elizaveta
664d6ab7d85a68a939cc73f7
Nikolai Gerasimenko
admin_assigned
2024-05-22T07:35:27.930Z
63cfa7ef3b7adfa99c0eb524
https://cdn-avatars.huggingface.co/v1/production/uploads/1674553277288-noauth.jpeg
Nikolai Gerasimenko
user
nikgerasimenko
664d6ab7d85a68a939cc73f8
Ivan Oseledets
admin_assigned
2024-05-22T07:35:33.932Z
6169a581d05945bfd8718dfa
/avatars/1892ab06a7ddb557232777de3cbec470.svg
Ivan Oseledets
user
oseledets
664d6ab7d85a68a939cc73f9
Denis Dimitrov
claimed_verified
2024-06-28T13:07:39.055Z
6669a678465d1d802181e456
https://cdn-avatars.huggingface.co/v1/production/uploads/6669a678465d1d802181e456/ZCthBBhDFQnh0bBkgUQUU.png
Denis Dimitrov
user
dendimitrov
664d6ab7d85a68a939cc73fa
Andrey Kuznetsov
claimed_verified
2024-05-22T08:55:57.659Z
643984dceb7c5616ef3f5d54
https://cdn-avatars.huggingface.co/v1/production/uploads/643984dceb7c5616ef3f5d54/10JRkblrRIEVci6UJwvPz.jpeg
Andrey Kuznetsov
user
kuznetsoffandrey
Your Transformer is Secretly Linear
This paper reveals a novel linear characteristic exclusive to transformer
decoders, including models such as GPT, LLaMA, OPT, BLOOM and others. We
analyze embedding transformations between sequential layers, uncovering a
near-perfect linear relationship (Procrustes similarity score of 0.99).
However, linearity decreases when the residual component is removed due to a
consistently low output norm of the transformer layer. Our experiments show
that removing or linearly approximating some of the most linear blocks of
transformers does not affect significantly the loss or model performance.
Moreover, in our pretraining experiments on smaller models we introduce a
cosine-similarity-based regularization, aimed at reducing layer linearity. This
regularization improves performance metrics on benchmarks like Tiny Stories and
SuperGLUE and as well successfully decreases the linearity of the models. This
study challenges the existing understanding of transformer architectures,
suggesting that their operation may be more linear than previously assumed.
Transformer decoders exhibit near-perfect linear relationships between layers, which can be reduced with cosine-similarity-based regularization, leading to improved performance on benchmarks.
transformer decoders Procrustes similarity score residual component linear blocks cosine-similarity-based regularization Tiny Stories SuperGLUE