Hankel Singular Value Regularization for Highly Compressible State Space Models
Paul Schwerdtner, Jules Berman, Benjamin Peherstorfer
Deep neural networks using state space models as layers are well suited for long-range sequence tasks but can be challenging to compress after training.We introduce Hankel singular value regularization (HSVR) that encourages decay in the Hankel singular values so that the state space models become compressible. To make the proposed regularization scalable, we develop an algorithm to efficiently compute the Hankel singular values during training iterations by exploiting the specific block-diagonal structure of the system matrices that is typically used in state space models. Experiments on Long Range Arena benchmarks demonstrate that HSVR state space layers are up to 10$\times$ more compressible than standard state space layers trained without regularization while maintaining high accuracy.