Efficient Training of Minimal and Maximal Low-Rank Recurrent Neural Networks
Anushri Arora, Jonathan Pillow
Low-rank recurrent neural networks (RNNs) provide a powerful framework for characterizing how neural systems solve complex cognitive tasks. However, fitting and interpreting these networks remains an important open problem. In this paper, we develop new methods for efficiently fitting low-rank RNNs in `teacher-training'' settings.  In particular, we build upon the neural engineering framework (NEF), in which RNNs are viewed as approximating an ordinary differential equation (ODE) of interest using a set of random nonlinear basis functions. This view provides geometric insight into how the choice of neural nonlinearity (e.g. tanh, ReLU) and the distribution of model parameters affects an RNN's representational capacity. Next, we adapt this framework for online training and demonstrate better performance with significantly smaller networks compared to FORCE. Additionally, we outperform backpropagation-trained networks of similar size—while requiring substantially less training time. Next, we ask: what basis functions—and how many—are needed to approximate a given dynamical system? To address this, we introduce methods for finding the smallest low-rank RNN to implement a dynamical system of interest using extensions of orthogonal matching pursuit. We then consider infinite unit low-rank RNNs, which converge to a Gaussian Process.  In particular, we show that we can optimize the distribution of nonlinear basis functions using maximum marginal likelihood for hyperparameters of the equivalent kernel function---which can be computed in closed form for particular choices of nonlinearity, resulting in substantially better generalization performance from limited data. Finally, we describe active learning methods for selecting highly informative datapoints that speed up training of low-rank RNNs.