Hyper-Connections
We present hyper-connections, a simple yet effective method that can serve as
an alternative to residual connections. This approach specifically addresses
common drawbacks observed in residual connection variants, such as the seesaw
effect between gradient vanishing and representation collapse. Theoretically,
hyper-connections allow the network to adjust the strength of connections
between features at different depths and dynamically rearrange layers. We
conduct experiments focusing on the pre-training of large language models,
including dense and sparse models, where hyper-connections show significant
performance improvements over residual connections. Additional experiments
conducted on vision tasks also demonstrate similar improvements. We anticipate
that this method will be broadly applicable and beneficial across a wide range
of AI problems.
66fb8def1c69086d311e8202
Defa Zhu
admin_assigned
2024-10-01T08:40:16.606Z
667505f4361b960c79e35486
/avatars/d352639c520075220f6abaae23c39376.svg
Defa Zhu
user
mathfinder
66fb8def1c69086d311e8203
Hongzhi Huang
admin_assigned
2024-10-01T08:40:21.956Z
66defe949fda73ea123ede1c
/avatars/52bda14a533fdd13436bcf7c65a38fb3.svg
huang
user
hongzhihuang
66fb8def1c69086d311e8204
Zihao Huang
claimed_verified
2024-10-10T09:10:58.219Z
65a62085576772f531e13856
/avatars/72c67a60422e333ea4e323f7480ae0b7.svg
Huang Zihao
user
FetchFortune
66fb8def1c69086d311e8205
Yutao Zeng
claimed_verified
2024-10-01T07:16:36.484Z
6371128eafbe42caa5a5222b
/avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
Yutao Zeng
user
Taoer
66fb8def1c69086d311e8206
Yunyao Mao
admin_assigned
2024-10-01T08:40:53.027Z
662b1e3b637672f56f81d4b7
/avatars/f5367089a6c69b3538bf8c9f8b411864.svg
Yunyao Mao
user
YunyaoMao
66fb8def1c69086d311e8207
Banggu Wu
admin_assigned
2024-10-01T08:41:05.200Z
64a3b0fa565496b629879293
/avatars/9d5de6cc4a01052ea971701f72bd3489.svg
wubanggu
user
banggu
66fb8def1c69086d311e8208
Qiyang Min
admin_assigned
2024-10-01T08:41:16.184Z
645507b89d37c3fb33279fe3
/avatars/ec1122137c49204ab968182d1f726c35.svg
min
user
qiyang-attn
66fb8def1c69086d311e8209
Xun Zhou
admin_assigned
2024-10-01T08:41:43.253Z
62533db4a06ec75172eeabe7
/avatars/b1a4dad90afae5c00df97233a97777db.svg
xunzhou
user
xunzhou
Hyper-Connections
We present hyper-connections, a simple yet effective method that can serve as
an alternative to residual connections. This approach specifically addresses
common drawbacks observed in residual connection variants, such as the seesaw
effect between gradient vanishing and representation collapse. Theoretically,
hyper-connections allow the network to adjust the strength of connections
between features at different depths and dynamically rearrange layers. We
conduct experiments focusing on the pre-training of large language models,
including dense and sparse models, where hyper-connections show significant
performance improvements over residual connections. Additional experiments
conducted on vision tasks also demonstrate similar improvements. We anticipate
that this method will be broadly applicable and beneficial across a wide range
of AI problems.
Hyper-connections, an alternative to residual connections, improve performance in pre-training large language models and vision tasks by dynamically adjusting feature connections.
hyper-connections residual connections seesaw effect gradient vanishing representation collapse pre-training large language models vision tasks