BranchNorm: Robustly Scaling Extremely Deep Transformers
Recently, DeepNorm scales Transformers into extremely deep (i.e., 1000
layers) and reveals the promising potential of deep scaling. To stabilize the
training of deep models, DeepNorm (Wang et al., 2022) attempts to constrain the
model update to a constant value. Although applying such a constraint can
benefit the early stage of model training, it may lead to undertrained models
during the whole training procedure. In this paper, we propose BranchNorm,
which dynamically rescales the non-residual branch of Transformer in accordance
with the training period. BranchNorm not only theoretically stabilizes the
training with smooth gradient norms at the early stage, but also encourages
better convergence in the subsequent training stage. Experiment results on
multiple translation tasks demonstrate that BranchNorm achieves a better
trade-off between training stability and converge performance.
64547a49a7ce0a8fde817551
Yijin Liu
64547a49a7ce0a8fde817552
Xianfeng Zeng
admin_assigned
2023-06-02T13:22:45.218Z
63c210b5a0ffa3857eb591ea
/avatars/7d663ea8e8dce3fa4ebe86cea758ef74.svg
zengxianfeng
user
zreading
64547a49a7ce0a8fde817553
Fandong Meng
64547a49a7ce0a8fde817554
Jie Zhou
BranchNorm: Robustly Scaling Extremely Deep Transformers
Recently, DeepNorm scales Transformers into extremely deep (i.e., 1000
layers) and reveals the promising potential of deep scaling. To stabilize the
training of deep models, DeepNorm (Wang et al., 2022) attempts to constrain the
model update to a constant value. Although applying such a constraint can
benefit the early stage of model training, it may lead to undertrained models
during the whole training procedure. In this paper, we propose BranchNorm,
which dynamically rescales the non-residual branch of Transformer in accordance
with the training period. BranchNorm not only theoretically stabilizes the
training with smooth gradient norms at the early stage, but also encourages
better convergence in the subsequent training stage. Experiment results on
multiple translation tasks demonstrate that BranchNorm achieves a better
trade-off between training stability and converge performance.
BranchNorm dynamically adjusts the scaling of the non-residual branch in Transformers to improve training stability and convergence in deep models.
DeepNorm Transformers deep scaling model update BranchNorm gradient norms training stability convergence