Recurrent Reinforcement Learning with Surrogate Gradients for Spiking Neural Networks
Korneel Van den Berghe, Stein Stroobants, Vijay Janapa Reddi, Guido de Croon
Neuromorphic computing systems are set to revolutionize energy-constrained robotics by achieving orders-of-magnitude efficiency gains while enabling native processing of temporal information. Spiking Neural Networks (SNNs) represent a promising algorithmic approach for these systems, yet their application to complex control tasks faces two critical challenges: (1) the non-differentiable nature of spiking neurons necessitates surrogate gradients with unclear optimization properties, and (2) the stateful dynamics of SNNs require training on sequences, which in reinforcement learning (RL) is hindered by limited sequence lengths during early training, preventing the network from bridging its warm-up period.We address these challenges by first providing a systematic analysis of surrogate gradient slope settings, demonstrating that shallower surrogate gradient slope settings increase gradient magnitude in deeper layers while reducing alignment with true gradients. In supervised learning, we find a slight preference of using steep slopes or scheduled slope settings, but the effect is much more pronounced in RL settings, where shallower slopes or scheduled slopes lead to a 2.1x improvement in both training performance and final deployed performance. Next, we propose a novel training approach that leverages a privileged guiding policy to bootstrap the learning process, while still exploiting online environment interactions. Combining our method with an adaptive slope schedule for a drone position control task, we achieve an average return of 400 points, substantially outperforming prior techniques, including behavioral cloning and TD3+BC, which achieve at most –200 points under the same conditions. This work advances both the theoretical understanding of surrogate gradients in SNNs and practical training methodologies for neuromorphic controllers demonstrated in real-world robotic systems.