Better separation and better NTK conditioning: effects of non-linear activation on wide neural networks
Chaoyue Liu, Han Bi, Like Hui, Xiao Liu
Non-linear activation functions are widely recognized for enhancing the expressivity of neural networks, which is the primary reason for their widespread implementation. In this work, we reveal a novel and intriguing property of non-linear activations. By comparing enabling and disabling the non-linear activations in the neural network, we demonstrate their specific effects on wide neural networks: (a) *better feature separation*, i.e.,  a larger angle separation for similar data in the feature space of model gradient, and (b) *better NTK conditioning*, i.e.,  a smaller condition number of neural tangent kernel (NTK). Furthermore, we show that the network depth (i.e., with more non-linear activation operations) further magnifies these effects; in addition, in the infinite-width-then-depth limit, all data are equally separated with a fixed angle in the model gradient feature space, regardless of how similar they are originally in the input space.    Note that, without the non-linear activation, i.e., in a linear neural network, the data separation remains the same as for the original inputs   and NTK condition number is equivalent to the Gram matrix, regardless of the network depth. Due to the close connection between NTK condition number and convergence theories, our results imply that non-linear activation  helps to improve the worst-case convergence rates of gradient based methods.