Value Residual Learning For Alleviating Attention Concentration In
  Transformers
Transformers can capture long-range dependencies using self-attention,
allowing tokens to attend to all others directly. However, stacking multiple
attention layers leads to attention concentration. One natural way to address
this issue is to use cross-layer attention, allowing information from earlier
layers to be directly accessible to later layers. However, this approach is
computationally expensive. To address this problem, we propose Transformer with
residual value (ResFormer) which approximates cross-layer attention through
adding a residual connection from the values of the the first layer to all
subsequent layers. Based on this method, one variant is the Transformer with
single layer value (SVFormer), where all layers share the same value embedding
from first layer, reducing the KV cache by nearly 50%. Comprehensive empirical
evidence demonstrates that ResFormer mitigates attention concentration problem
in deeper layers and enhances representation across most layers, outperforming
the vanilla Transformer, DenseFormer, and NeuTRENO in training error as well as
downstream tasks. SVFormer trains significantly faster than the vanilla
Transformer and performs better than other methods like GQA and CLA, with
performance influenced by sequence length and cumulative learning rate.
671b3e033fd1d03dc68d685e
Zhanchao Zhou
claimed_verified
2024-10-25T09:28:30.222Z
671b3d82101c0d9f78749256
/avatars/902c1006280c6552885b0b946cd2bb16.svg
Zhanchao Zhou
user
Zcchill
671b3e033fd1d03dc68d685f
Tianyi Wu
671b3e033fd1d03dc68d6860
Zhiyun Jiang
671b3e033fd1d03dc68d6861
Zhenzhong Lan
Value Residual Learning For Alleviating Attention Concentration In
  Transformers
Transformers can capture long-range dependencies using self-attention,
allowing tokens to attend to all others directly. However, stacking multiple
attention layers leads to attention concentration. One natural way to address
this issue is to use cross-layer attention, allowing information from earlier
layers to be directly accessible to later layers. However, this approach is
computationally expensive. To address this problem, we propose Transformer with
residual value (ResFormer) which approximates cross-layer attention through
adding a residual connection from the values of the the first layer to all
subsequent layers. Based on this method, one variant is the Transformer with
single layer value (SVFormer), where all layers share the same value embedding
from first layer, reducing the KV cache by nearly 50%. Comprehensive empirical
evidence demonstrates that ResFormer mitigates attention concentration problem
in deeper layers and enhances representation across most layers, outperforming
the vanilla Transformer, DenseFormer, and NeuTRENO in training error as well as
downstream tasks. SVFormer trains significantly faster than the vanilla
Transformer and performs better than other methods like GQA and CLA, with
performance influenced by sequence length and cumulative learning rate.
ResFormer and SVFormer variants of Transformers improve long-range dependency capture and reduce computational cost by introducing residual connections and shared value embeddings.
self-attention cross-layer attention residual value SVFormer DenseFormer NeuTRENO GQA CLA