Smooth Regularization for Efficient Video Recognition
Gil Goldman, Raja Giryes, Mahadev Satyanarayanan
We propose a smooth regularization technique that instillsa strong temporal inductive bias in video recognition models,particularly benefiting lightweight architectures.Our method encourages smoothness in the intermediate-layer embeddings ofconsecutive frames by modeling their changes as a Gaussian Random Walk (GRW).This penalizes abrupt representational shifts,thereby promoting low-acceleration solutions that better alignwith the natural temporal coherence inherent in videos.By leveraging this enforced smoothness, lightweight models can moreeffectively capture complex temporal dynamics.Applied to such models, our technique yields a 2.8%–6.0%accuracy improvement on Kinetics-600.Notably, the MoViNets model family trained with our smoothregularization improves the current state-of-the-art by 2.8%–3.4% withintheir respective FLOP constraints, while MobileNetV3 andthe MoViNets-Stream family achieve gains of 2.8%–6.0% overprior state-of-the-art models with comparable memory footprints.Code and models will be released upon acceptance.