MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway
  Dynamic Dense Connections
We propose MUltiway Dynamic Dense (MUDD) connections, a simple yet effective
method to address the limitations of residual connections and enhance
cross-layer information flow in Transformers. Unlike existing dense connection
approaches with static and shared connection weights, MUDD generates connection
weights dynamically depending on hidden states at each sequence position and
for each decoupled input stream (the query, key, value or residual) of a
Transformer block. MUDD connections can be seamlessly integrated into any
Transformer architecture to create MUDDFormer. Extensive experiments show that
MUDDFormer significantly outperforms Transformers across various model
architectures and scales in language modeling, achieving the performance of
Transformers trained with 1.8X-2.4X compute. Notably, MUDDPythia-2.8B matches
Pythia-6.9B in pretraining ppl and downstream tasks and even rivals Pythia-12B
in five-shot settings, while adding only 0.23% parameters and 0.4% computation.
Code in JAX and PyTorch and pre-trained models are available at
https://github.com/Caiyun-AI/MUDDFormer .
67b5434f2b2ec6908fffe75e
Da Xiao
67b5434f2b2ec6908fffe75f
Qingye Meng
admin_assigned
2025-02-19T10:23:45.040Z
634f9c94cdc89e42cc7b194a
/avatars/42376474afe9b68dc44184c71e210529.svg
Qingye Meng
user
Hilbertmeng
67b5434f2b2ec6908fffe760
Shengping Li
admin_assigned
2025-02-19T10:23:38.331Z
6706a37cca9b1a88fc3951ea
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Y4GvpKiVgqm_2Kod52_B8.png
lishengping
user
lishengping
67b5434f2b2ec6908fffe761
Xingyuan Yuan
MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway
  Dynamic Dense Connections
We propose MUltiway Dynamic Dense (MUDD) connections, a simple yet effective
method to address the limitations of residual connections and enhance
cross-layer information flow in Transformers. Unlike existing dense connection
approaches with static and shared connection weights, MUDD generates connection
weights dynamically depending on hidden states at each sequence position and
for each decoupled input stream (the query, key, value or residual) of a
Transformer block. MUDD connections can be seamlessly integrated into any
Transformer architecture to create MUDDFormer. Extensive experiments show that
MUDDFormer significantly outperforms Transformers across various model
architectures and scales in language modeling, achieving the performance of
Transformers trained with 1.8X-2.4X compute. Notably, MUDDPythia-2.8B matches
Pythia-6.9B in pretraining ppl and downstream tasks and even rivals Pythia-12B
in five-shot settings, while adding only 0.23% parameters and 0.4% computation.
Code in JAX and PyTorch and pre-trained models are available at
https://github.com/Caiyun-AI/MUDDFormer .
MUDD connections dynamically generate weights for cross-layer information flow in Transformers, enhancing performance with minimal computational overhead.
MUltiway Dynamic Dense MUDD residual connections dense connection hidden states sequence position decoupled input stream Transformer block MUDDFormer language modeling pretraining ppl downstream tasks five-shot settings JAX PyTorch pre-trained models