Reproducing Kernel Banach Space Models for Neural Networks with Application to Rademacher Complexity Analysis
Alistair Shilton, Sunil Gupta, Santu Rana, Svetha Venkatesh
This paper explores the use of Hermite transform based reproducing kernel Banach space methods to construct exact or un-approximated models of feedforward neural networks of arbitrary width, depth and topology, including residual networks, transformers and other attention-based networks, assuming only a feedforward topology, finite energy activations and finite (spectral-) norm weights and biases. Using this model, two straightforward but surprisingly tight bounds on Rademacher complexity are derived, precisely (1) a general bound that is width-independent and scales exponentially with depth; and (2) a width- and depth-independent bound for networks (including ReLU, ResNet and transformer networks) with appropriately constrained (below threshold) weights and biases.