Elastic ViTs from Pretrained Models without Retraining
Walter Simoncini, Michael Dorkenwald, Tijmen Blankevoort, Cees Snoek, Yuki Asano
Vision foundation models achieve remarkable performance but are only available in a limited set of pre-determined sizes, forcing sub-optimal deployment choices under real-world constraints. We introduce a novel post-pretraining structured pruning method that enables elastic inference across a continuum of compute budgets. Our approach combines gradient information with cross-network structure correlations, efficiently approximated through an evolutionary algorithm, does not require labeled data, generalizes to models without a classification head, and is retraining free. Experiments on DINO and AugReg models demonstrate superior performance over state of the art methods across various sparsities, requiring less than five minutes on a A100 GPU to generate elastic models that can be adjusted to any computational budget. Our key contributions include an efficient pruning strategy for pretrained Vision Transformers, a novel evolutionary approximation of Hessian off-diagonal structures, and a self-supervised importance scoring mechanism that maintains strong performance without requiring retraining nor labels.