ModHiFi: Identifying High Fidelity predictive components for Model Modification
Dhruva Kashyap, Chaitanya Murti, Tanay Narshana, Pranav Nayak, Chiranjib Bhattacharyya
Modifying well-trained models for tasks such as pruning or unlearning without access to training data or the original loss function is a challenging problem. While various techniques exist for tasks like pruning and classwise unlearning, they often require training data, are computationally expensive, or are architecture-specific. To address this, we investigate the fundamental question of identifying critical model components that significantly influence predictive performance without using gradients. Contrary to existing literature, we empirically observe that well-trained Transformer models exhibit Lipschitz continuity. Leveraging this insight, we theoretically demonstrate that for Lipschitz networks, the global reconstruction error is linearly bounded by local reconstruction errors for CNNs and well-trained transformers. This motivates our introduction of \emph{Subset Fidelity}, a novel metric to quantify the importance of component subsets by assessing their ability to reconstruct model outputs after weight adjustments. We further show that for uncorrelated features, selecting components with the highest individual Subset Fidelity scores is optimal. Based on this, we propose {\bf ModHiFi}, an algorithm for model modification that does not require the training data or loss function. Specifically, ModHiFi-P for structured pruning achieves 11% speed up improvement over current state of the art on ImageNet models and competitive performance on language models. For classwise unlearning, ModHiFi-U achieves complete unlearning on CIFAR-10 without fine-tuning and demonstrates competitive performance on Swin-Transformers.