Adapting Frame-Based Networks for Stable and Robust Video Inference
Matthew Dutson, Nathan Labiosa, Yin Li, Mohit Gupta
Frame-based neural networks often exhibit temporal inconsistency, i.e., flickering or incoherent predictions, when applied to videos. This problem is amplified when the input frames contain time-varying corruptions such as noise. Many prior works propose stable video-optimized networks for individual tasks or groups of related tasks. In this work, we introduce a general approach for adapting frame-wise networks for stable inference on video data. We describe a class of stabilization adapters that can be inserted into virtually any architecture and a resource-efficient training process that can be performed with a frozen base network. We introduce a unified conceptual framework for describing temporal stability and corruption robustness, centered on a proposed accuracy-stability-robustness loss. We analyze the theoretical properties of this loss, identifying the conditions where it produces well-behaved training. Our experiments validate our approach on several vision tasks including denoising (NAFNet), image enhancement (HDRNet), and monocular depth (Depth Anything v2). Our method improves temporal stability and robustness against a range of image corruptions, while generally preserving or improving task accuracy.