The Curse of Depth in Large Language Models
In this paper, we introduce the Curse of Depth, a concept that highlights,
explains, and addresses the recent observation in modern Large Language
Models(LLMs) where nearly half of the layers are less effective than expected.
We first confirm the wide existence of this phenomenon across the most popular
families of LLMs such as Llama, Mistral, DeepSeek, and Qwen. Our analysis,
theoretically and empirically, identifies that the underlying reason for the
ineffectiveness of deep layers in LLMs is the widespread usage of Pre-Layer
Normalization (Pre-LN). While Pre-LN stabilizes the training of Transformer
LLMs, its output variance exponentially grows with the model depth, which
undesirably causes the derivative of the deep Transformer blocks to be an
identity matrix, and therefore barely contributes to the training. To resolve
this training pitfall, we propose LayerNorm Scaling, which scales the variance
of output of the layer normalization inversely by the square root of its depth.
This simple modification mitigates the output variance explosion of deeper
Transformer layers, improving their contribution. Our experimental results,
spanning model sizes from 130M to 1B, demonstrate that LayerNorm Scaling
significantly enhances LLM pre-training performance compared to Pre-LN.
Moreover, this improvement seamlessly carries over to supervised fine-tuning.
All these gains can be attributed to the fact that LayerNorm Scaling enables
deeper layers to contribute more effectively during training.
67ab189a8087b66340398b01
Wenfang Sun
claimed_verified
2025-02-11T14:25:29.711Z
643ce831a16fa581f3f826c9
/avatars/2f2dffb660eee3d3c7029dd7305f5226.svg
Wenfang Sun
user
lmsdss
67ab189a8087b66340398b02
Xinyuan Song
claimed_verified
2025-02-13T20:36:06.467Z
6728a79b2a2e92f37afb900d
/avatars/92106187ccbe34dfdbfc5d6d6fd63210.svg
XinyuanSong
user
XinyuanSong
67ab189a8087b66340398b03
Pengxiang Li
claimed_verified
2025-02-11T09:51:15.671Z
64245f2c089d5fae56b4549a
https://cdn-avatars.huggingface.co/v1/production/uploads/64245f2c089d5fae56b4549a/qUHFsL9Svwyj5BKpfMtaY.jpeg
Pengxiang Li
user
pengxiang
67ab189a8087b66340398b04
Lu Yin
67ab189a8087b66340398b05
Yefeng Zheng
67ab189a8087b66340398b06
Shiwei Liu
claimed_verified
2025-03-03T11:15:25.316Z
65b04d2291e63920a7898c9e
https://cdn-avatars.huggingface.co/v1/production/uploads/65b04d2291e63920a7898c9e/iUHs235G4bqK-KnH_94ti.jpeg
Liu
user
Shiweiliuiiiiiii
The Curse of Depth in Large Language Models
In this paper, we introduce the Curse of Depth, a concept that highlights,
explains, and addresses the recent observation in modern Large Language
Models(LLMs) where nearly half of the layers are less effective than expected.
We first confirm the wide existence of this phenomenon across the most popular
families of LLMs such as Llama, Mistral, DeepSeek, and Qwen. Our analysis,
theoretically and empirically, identifies that the underlying reason for the
ineffectiveness of deep layers in LLMs is the widespread usage of Pre-Layer
Normalization (Pre-LN). While Pre-LN stabilizes the training of Transformer
LLMs, its output variance exponentially grows with the model depth, which
undesirably causes the derivative of the deep Transformer blocks to be an
identity matrix, and therefore barely contributes to the training. To resolve
this training pitfall, we propose LayerNorm Scaling, which scales the variance
of output of the layer normalization inversely by the square root of its depth.
This simple modification mitigates the output variance explosion of deeper
Transformer layers, improving their contribution. Our experimental results,
spanning model sizes from 130M to 1B, demonstrate that LayerNorm Scaling
significantly enhances LLM pre-training performance compared to Pre-LN.
Moreover, this improvement seamlessly carries over to supervised fine-tuning.
All these gains can be attributed to the fact that LayerNorm Scaling enables
deeper layers to contribute more effectively during training.
LayerNorm Scaling addresses the Curse of Depth in LLMs by scaling variance inversely with layer depth, improving pre-training and fine-tuning effectiveness of deep layers.
Curse of Depth Large Language Models Pre-Layer Normalization Pre-LN Transformer output variance derivative training LayerNorm Scaling pre-training supervised fine-tuning