Transformer Layers as Painters
Despite their nearly universal adoption for large language models, the
internal workings of transformers are not well understood. We aim to better
understand the impact of removing or reorganizing information throughout the
layers of a pretrained transformer. Such an understanding could both yield
better usage of existing models as well as to make architectural improvements
to produce new variants. We present a series of empirical studies on frozen
models that show that the lower and final layers of pretrained transformers
differ from middle layers, but that middle layers have a surprising amount of
uniformity. We further show that some classes of problems have robustness to
skipping layers, running the layers in an order different from how they were
trained, or running the layers in parallel. Our observations suggest that even
frozen pretrained models may gracefully trade accuracy for latency by skipping
layers or running layers in parallel.
669489afcab4aab691e57e1a
Qi Sun
claimed_verified
2024-08-05T09:02:32.335Z
644b983f0fbe4830f192c4f5
https://cdn-avatars.huggingface.co/v1/production/uploads/644b983f0fbe4830f192c4f5/4eX4OS0gnXGqLu3eaMml_.png
Qi Sun
user
lfsm
669489afcab4aab691e57e1b
Marc Pickett
669489afcab4aab691e57e1c
Aakash Kumar Nain
669489afcab4aab691e57e1d
Llion Jones
Transformer Layers as Painters
Despite their nearly universal adoption for large language models, the
internal workings of transformers are not well understood. We aim to better
understand the impact of removing or reorganizing information throughout the
layers of a pretrained transformer. Such an understanding could both yield
better usage of existing models as well as to make architectural improvements
to produce new variants. We present a series of empirical studies on frozen
models that show that the lower and final layers of pretrained transformers
differ from middle layers, but that middle layers have a surprising amount of
uniformity. We further show that some classes of problems have robustness to
skipping layers, running the layers in an order different from how they were
trained, or running the layers in parallel. Our observations suggest that even
frozen pretrained models may gracefully trade accuracy for latency by skipping
layers or running layers in parallel.
Empirical studies on frozen transformers reveal differences between lower/final and middle layers and suggest that these models can maintain robustness and trade accuracy for latency by skipping or rearranging layers.
transformers frozen models lower layers final layers middle layers skipping layers parallel layers