Frac-Connections: Fractional Extension of Hyper-Connections
Residual connections are central to modern deep learning architectures,
enabling the training of very deep networks by mitigating gradient vanishing.
Hyper-Connections recently generalized residual connections by introducing
multiple connection strengths at different depths, thereby addressing the
seesaw effect between gradient vanishing and representation collapse. However,
Hyper-Connections increase memory access costs by expanding the width of hidden
states. In this paper, we propose Frac-Connections, a novel approach that
divides hidden states into multiple parts rather than expanding their width.
Frac-Connections retain partial benefits of Hyper-Connections while reducing
memory consumption. To validate their effectiveness, we conduct large-scale
experiments on language tasks, with the largest being a 7B MoE model trained on
up to 3T tokens, demonstrating that Frac-Connections significantly outperform
residual connections.
67da200db41738a058666623
Defa Zhu
admin_assigned
2025-03-19T13:32:01.715Z
667505f4361b960c79e35486
/avatars/d352639c520075220f6abaae23c39376.svg
Defa Zhu
user
mathfinder
67da200db41738a058666624
Hongzhi Huang
admin_assigned
2025-03-19T13:32:07.284Z
66defe949fda73ea123ede1c
/avatars/52bda14a533fdd13436bcf7c65a38fb3.svg
huang
user
hongzhihuang
67da200db41738a058666625
Jundong Zhou
admin_assigned
2025-03-19T13:32:22.854Z
642ae9f4d651bae3c11ebad1
/avatars/d0f9253da54e148c152710923d8d3804.svg
Zhou Jundong
user
ulrical
67da200db41738a058666626
Zihao Huang
claimed_verified
2025-03-19T09:45:07.454Z
65a62085576772f531e13856
/avatars/72c67a60422e333ea4e323f7480ae0b7.svg
Huang Zihao
user
FetchFortune
67da200db41738a058666627
Yutao Zeng
claimed_verified
2025-03-19T09:45:05.382Z
6371128eafbe42caa5a5222b
/avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
Yutao Zeng
user
Taoer
67da200db41738a058666628
Banggu Wu
admin_assigned
2025-03-19T13:32:34.670Z
64a3b0fa565496b629879293
/avatars/9d5de6cc4a01052ea971701f72bd3489.svg
wubanggu
user
banggu
67da200db41738a058666629
Qiyang Min
admin_assigned
2025-03-19T13:32:46.730Z
645507b89d37c3fb33279fe3
/avatars/ec1122137c49204ab968182d1f726c35.svg
min
user
qiyang-attn
67da200db41738a05866662a
Xun Zhou
Frac-Connections: Fractional Extension of Hyper-Connections
Residual connections are central to modern deep learning architectures,
enabling the training of very deep networks by mitigating gradient vanishing.
Hyper-Connections recently generalized residual connections by introducing
multiple connection strengths at different depths, thereby addressing the
seesaw effect between gradient vanishing and representation collapse. However,
Hyper-Connections increase memory access costs by expanding the width of hidden
states. In this paper, we propose Frac-Connections, a novel approach that
divides hidden states into multiple parts rather than expanding their width.
Frac-Connections retain partial benefits of Hyper-Connections while reducing
memory consumption. To validate their effectiveness, we conduct large-scale
experiments on language tasks, with the largest being a 7B MoE model trained on
up to 3T tokens, demonstrating that Frac-Connections significantly outperform
residual connections.
Frac-Connections, a technique that divides hidden states, outperform traditional residual connections in deep learning by balancing memory consumption and performance, validated through experiments on large language models.
residual connections gradient vanishing Hyper-Connections representation collapse Frac-Connections hidden states memory consumption large-scale experiments language tasks 7B MoE model 3T tokens