Revisiting Residual Connections: Orthogonal Updates for Stable and
  Efficient Deep Networks
Residual connections are pivotal for deep neural networks, enabling greater
depth by mitigating vanishing gradients. However, in standard residual updates,
the module's output is directly added to the input stream. This can lead to
updates that predominantly reinforce or modulate the existing stream direction,
potentially underutilizing the module's capacity for learning entirely novel
features. In this work, we introduce Orthogonal Residual Update: we decompose
the module's output relative to the input stream and add only the component
orthogonal to this stream. This design aims to guide modules to contribute
primarily new representational directions, fostering richer feature learning
while promoting more efficient training. We demonstrate that our orthogonal
update strategy improves generalization accuracy and training stability across
diverse architectures (ResNetV2, Vision Transformers) and datasets (CIFARs,
TinyImageNet, ImageNet-1k), achieving, for instance, a +4.3\%p top-1 accuracy
gain for ViT-B on ImageNet-1k.
6833ebdb142b0e50399413d3
Giyeong Oh
claimed_verified
2025-05-28T09:01:02.174Z
63d93667255ef6add20f9272
/avatars/99a3aeadcc81ef85164cdfb6ab186b17.svg
Giyeong Oh
user
BootsofLagrangian
6833ebdb142b0e50399413d4
Woohyun Cho
6833ebdb142b0e50399413d5
Siyeol Kim
claimed_verified
2025-05-28T09:01:04.269Z
67021743e4d49b157afd8260
/avatars/2a22a18cd45f6d115e8a3a5d1e477dcb.svg
Siyeol Kim
user
siyeolkim
6833ebdb142b0e50399413d6
Suhwan Choi
6833ebdb142b0e50399413d7
Younjae Yu
Revisiting Residual Connections: Orthogonal Updates for Stable and
  Efficient Deep Networks
Residual connections are pivotal for deep neural networks, enabling greater
depth by mitigating vanishing gradients. However, in standard residual updates,
the module's output is directly added to the input stream. This can lead to
updates that predominantly reinforce or modulate the existing stream direction,
potentially underutilizing the module's capacity for learning entirely novel
features. In this work, we introduce Orthogonal Residual Update: we decompose
the module's output relative to the input stream and add only the component
orthogonal to this stream. This design aims to guide modules to contribute
primarily new representational directions, fostering richer feature learning
while promoting more efficient training. We demonstrate that our orthogonal
update strategy improves generalization accuracy and training stability across
diverse architectures (ResNetV2, Vision Transformers) and datasets (CIFARs,
TinyImageNet, ImageNet-1k), achieving, for instance, a +4.3\%p top-1 accuracy
gain for ViT-B on ImageNet-1k.
Orthogonal Residual Updates enhance feature learning and training stability by decomposing module outputs to contribute primarily novel features.
residual connections vanishing gradients orthogonal update ResNetV2 Vision Transformers CIFARs TinyImageNet ImageNet-1k top-1 accuracy