Parameter Efficient Merging for Multimodal Large Language Models with Direction Robustness
Fanhu Zeng, Haiyang Guo, Fei Zhu, Li Shen, Hao Tang
Fine-tuning pre-trained models with custom data leads to numerous expert models on specific tasks. Merging models into one universal model to empower multi-task ability refraining from data leakage has gained popularity.  With the expansion in data and model size, parameter efficient tuning becomes the common practice for obtaining task-specific models efficiently. However, few methods are dedicated to efficient merging and existing methods designed for full fine-tuning merging fail under efficient merging. To address the issue, we analyze from low-rank decomposition and reveal that direction robustness during merging is crucial for merging efficient modules. We furthermore uncover that compensating for gap between stark singular values contributes to direction robustness. Therefore, we propose CoPA-Merging, a training-free parameter efficient merging method with complementary parameter adaptation to maintain directions. Specifically, we (1) prune parameters and scale coefficients from inter-parameter relation for singular values to maintain direction stability away from task interference and (2) perform cross-task normalization to enhance unseen task generalization. We establish a benchmark consisting of diverse multimodal tasks, on which we conduct experiments to certificate the outstanding performance and generalizability of our method. Additional study and extensive analyses further showcase the effectiveness.