Boosting Machine Unlearning via Ensembling Infinite Number of Functions on Task Arithmetic Simplex
Junhao Dong, Hao Zhu, Yifei Zhang, Xinghua Qu, Yew Soon Ong, Piotr Koniusz
As foundation Vision-Language Models (VLMs) unlock fine-tuning on smaller datasets while capturing large-scale data in pre-training, machine unlearning becomes critical in addressing privacy concerns and regulatory compliance. Task vector, representing differences between parameters of models fine-tuned with and without specific data, is a popular retraining-free unlearning strategy. However, we observe that task vectors exhibit substantial sensitivity to various fine-tuning configurations, resulting in unstable unlearning effectiveness negatively correlating with prediction-level variance. While aggregating multiple functions (e.g., VLM with classifier) whose parameters are represented by different task vectors naturally reduces function variance and improves unlearning, the computational cost of obtaining numerous task vectors and aggregating functions is computationally high. Thus, to robustly capture the space of task vectors induced by diverse fine-tuning strategies, we propose modeling it within the convex hull of $(Q-1)$-simplex whose vertices are $Q$ task vectors. Although a function ensemble can be formed by sampling numerous task vectors from such a simplex, this is computationally prohibitive. Thus, we derive a closed-form ensemble of an infinite number of functions whose parameters are uniformly sampled from the simplex, enabling efficient function-level task vector ensembling with enhanced unlearning performance. Extensive experiments and analyses across diverse datasets and scenarios demonstrate the efficacy of our method.