Mechanistic Interpretability of RNNs emulating Hidden Markov Models
Elia Torre, Michele Viscione, Lucas Pompe, Benjamin F. Grewe, Valerio Mante
Recurrent neural networks (RNNs) provide a powerful approach in neuroscience to infer latent dynamics in neural populations and to generate hypotheses about the neural computations underlying behavior. However, past work has focused on relatively simple, input-driven, and largely deterministic behaviors - little is known about the mechanisms that would allow RNNs to generate the richer, spontaneous, and potentially stochastic behaviors observed in natural settings. Modeling with Hidden Markov Models (HMMs) has revealed a segmentation of natural behaviors into discrete latent states with stochastic transitions between them, a type of dynamics that may appear at odds with the continuous state spaces implemented by RNNs. Here we first show that RNNs can replicate HMM emission statistics and then reverse-engineer the trained networks to uncover the mechanisms they implement. In the absence of inputs, RNN activity collapses towards a single, stable fixed point. When driven by stochastic input, trajectories instead move along a noise-sustained limit cycle. Rotation along the cycle leads to a change in the emission probabilities, and is governed by transitions between regions of slow, noisy dynamics that are connected by fast, deterministic dynamics. The RNNs learn highly structured connectivity, with only a few 'kick neurons' initiating transitions between slow regions. This mechanism emerges through training, triggered by a Hopf bifurcation that shifts RNNs into a regime of stochastic resonance to implement probabilistic computations. This work demonstrates the potential of RNNs in modeling computations underlying spontaneous behaviors and leads to novel hypotheses about the role of stochastic resonance in biological neural networks.