GPAS: Accelerating Convergence of LLM Pretraining via
  Gradient-Preserving Activation Scaling
Modern Large Language Models, such as the LLaMA, Qwen and DeepSeek series,
predominantly adopt the Pre-LayerNorm (Pre-LN) Transformer architecture. While
being stable during pretraining and scalable to large model sizes, Pre-LN
suffers from an exponential growth in activation variance across layers,
causing the residual path to dominate over sub-layer outputs and limiting the
learning capacity of deeper layers. To mitigate this issue, we propose
Gradient-Preserving Activation Scaling (GPAS), a simple technique that can be
used in combination with existing approaches. GPAS works by scaling down the
intermediate activations while keeping their gradients unchanged. This leaves
information in the activations intact, and avoids the gradient vanishing
problem associated with gradient downscaling. Extensive experiments across
various model sizes from 71M to 1B show that GPAS achieves consistent
performance gains. Beyond enhancing Pre-LN Transformers, GPAS also shows
promise in improving alternative architectures such as Sandwich-LN and
DeepNorm, demonstrating its versatility and potential for improving training
dynamics in a wide range of settings.
6862a0ce3b7fc785c7ca5dea
Tianhao Chen
6862a0ce3b7fc785c7ca5deb
Xin Xu
claimed_verified
2025-09-26T12:25:01.241Z
64e2d169d2af12910d682130
https://cdn-avatars.huggingface.co/v1/production/uploads/64e2d169d2af12910d682130/VG8UdqJCJGc0K4G0P0XQP.jpeg
xuxin
user
xx18
6862a0ce3b7fc785c7ca5dec
Zijing Liu
6862a0ce3b7fc785c7ca5ded
Pengxiang Li
claimed_verified
2025-09-26T12:25:04.222Z
64245f2c089d5fae56b4549a
https://cdn-avatars.huggingface.co/v1/production/uploads/64245f2c089d5fae56b4549a/qUHFsL9Svwyj5BKpfMtaY.jpeg
Pengxiang Li
user
pengxiang
6862a0ce3b7fc785c7ca5dee
Xinyuan Song
6862a0ce3b7fc785c7ca5def
Ajay Kumar Jaiswal
6862a0ce3b7fc785c7ca5df0
Fan Zhang
6862a0ce3b7fc785c7ca5df1
Jishan Hu
6862a0ce3b7fc785c7ca5df2
Yang Wang
6862a0ce3b7fc785c7ca5df3
Hao Chen
6862a0ce3b7fc785c7ca5df4
Shizhe Diao
6862a0ce3b7fc785c7ca5df5
Shiwei Liu
6862a0ce3b7fc785c7ca5df6
Yu Li
6862a0ce3b7fc785c7ca5df7
Yin Lu
6862a0ce3b7fc785c7ca5df8
Can Yang
GPAS: Accelerating Convergence of LLM Pretraining via
  Gradient-Preserving Activation Scaling
Modern Large Language Models, such as the LLaMA, Qwen and DeepSeek series,
predominantly adopt the Pre-LayerNorm (Pre-LN) Transformer architecture. While
being stable during pretraining and scalable to large model sizes, Pre-LN
suffers from an exponential growth in activation variance across layers,
causing the residual path to dominate over sub-layer outputs and limiting the
learning capacity of deeper layers. To mitigate this issue, we propose
Gradient-Preserving Activation Scaling (GPAS), a simple technique that can be
used in combination with existing approaches. GPAS works by scaling down the
intermediate activations while keeping their gradients unchanged. This leaves
information in the activations intact, and avoids the gradient vanishing
problem associated with gradient downscaling. Extensive experiments across
various model sizes from 71M to 1B show that GPAS achieves consistent
performance gains. Beyond enhancing Pre-LN Transformers, GPAS also shows
promise in improving alternative architectures such as Sandwich-LN and
DeepNorm, demonstrating its versatility and potential for improving training
dynamics in a wide range of settings.
Gradient-Preserving Activation Scaling (GPAS) mitigates activation variance issues in Pre-LayerNorm Transformers and enhances training dynamics across different architectures.
Pre-LayerNorm Transformer architecture activation variance residual path learning capacity Gradient-Preserving Activation Scaling GPAS gradient vanishing problem Sandwich-LN DeepNorm training dynamics