You Do Not Fully Utilize Transformer's Representation Capacity
In contrast to RNNs, which compress previous tokens into a single hidden
state, Transformers can attend to all previous tokens directly. However,
standard Transformers only use representations from the immediately preceding
layer. In this paper, we show that this design choice causes representation
collapse and leads to suboptimal performance. To address this issue, we
introduce Layer-Integrated Memory (LIMe), a simple yet powerful approach that
preserves the model's overall memory footprint while expanding its
representational capacity by allowing access to hidden states from earlier
layers. Through extensive experiments across various architectures and
different lookup mechanisms, we demonstrate consistent performance improvements
on a wide range of tasks. Moreover, our analysis of the learned representation
dynamics and our exploration of depthwise circuits reveal how LIMe integrates
information across layers, pointing to promising directions for future
research.
67b57a993d4f319f1fa9424b
Gleb Gerasimov
claimed_verified
2025-02-19T10:10:30.547Z
65db0871ab2f64915ce05e73
/avatars/77e03f493196c5413cd2a02270e93660.svg
Gleb Gerasimov
user
gudleifrr
67b57a993d4f319f1fa9424c
Yaroslav Aksenov
claimed_verified
2025-02-19T09:00:41.123Z
63ed5676684767daecac6f8a
/avatars/d0e4a715f9c3fb6d74c183bab751ec35.svg
Yaroslav Aksenov
user
yaraksen
67b57a993d4f319f1fa9424d
Nikita Balagansky
claimed_verified
2025-02-19T09:33:26.858Z
60b364e7f88532cd79eaff7b
https://cdn-avatars.huggingface.co/v1/production/uploads/1654185363389-60b364e7f88532cd79eaff7b.jpeg
Nikita Balagansky
user
elephantmipt
67b57a993d4f319f1fa9424e
Viacheslav Sinii
claimed_verified
2025-02-19T11:12:28.927Z
6416272d986557e8cac64ece
https://cdn-avatars.huggingface.co/v1/production/uploads/6416272d986557e8cac64ece/s3CLjNN_pGj-vJDcENFD2.jpeg
Viacheslav
user
ummagumm-a
67b57a993d4f319f1fa9424f
Daniil Gavrilov
claimed_verified
2025-02-19T09:00:43.143Z
62a9c8edc19f92ae443ab37f
https://cdn-avatars.huggingface.co/v1/production/uploads/1669110208492-62a9c8edc19f92ae443ab37f.png
Daniil Gavrilov
user
kefirski
You Do Not Fully Utilize Transformer's Representation Capacity
In contrast to RNNs, which compress previous tokens into a single hidden
state, Transformers can attend to all previous tokens directly. However,
standard Transformers only use representations from the immediately preceding
layer. In this paper, we show that this design choice causes representation
collapse and leads to suboptimal performance. To address this issue, we
introduce Layer-Integrated Memory (LIMe), a simple yet powerful approach that
preserves the model's overall memory footprint while expanding its
representational capacity by allowing access to hidden states from earlier
layers. Through extensive experiments across various architectures and
different lookup mechanisms, we demonstrate consistent performance improvements
on a wide range of tasks. Moreover, our analysis of the learned representation
dynamics and our exploration of depthwise circuits reveal how LIMe integrates
information across layers, pointing to promising directions for future
research.
Layer-Integrated Memory (LIMe) addresses representation collapse in Transformers by allowing access to hidden states from earlier layers, leading to consistent performance improvements across tasks.
Transformers RNNs hidden state representation collapse Layer-Integrated Memory LIMe learned representation dynamics depthwise circuits