No Task Left Behind: Isotropic Model Merging with Common and
  Task-Specific Subspaces
Model merging integrates the weights of multiple task-specific models into a
single multi-task model. Despite recent interest in the problem, a significant
performance gap between the combined and single-task models remains. In this
paper, we investigate the key characteristics of task matrices -- weight update
matrices applied to a pre-trained model -- that enable effective merging. We
show that alignment between singular components of task-specific and merged
matrices strongly correlates with performance improvement over the pre-trained
model. Based on this, we propose an isotropic merging framework that flattens
the singular value spectrum of task matrices, enhances alignment, and reduces
the performance gap. Additionally, we incorporate both common and task-specific
subspaces to further improve alignment and performance. Our proposed approach
achieves state-of-the-art performance across multiple scenarios, including
various sets of tasks and model scales. This work advances the understanding of
model merging dynamics, offering an effective methodology to merge models
without requiring additional training. Code is available at
https://github.com/danielm1405/iso-merging .
67a9f4900b97667e0a82ad3d
Daniel Marczak
claimed_verified
2025-02-10T13:11:49.393Z
65a5358ddb5c00652ef24c8d
/avatars/d50b6297584c9b4c2ccd93e64477b940.svg
Daniel Marczak
user
danielm1405
67a9f4900b97667e0a82ad3e
Simone Magistri
67a9f4900b97667e0a82ad3f
Sebastian Cygert
admin_assigned
2025-02-10T16:17:02.946Z
6763f031c0a39e58c57ed9f9
/avatars/2e864838a571f52b5316f90d60b763f1.svg
Sebastian Cygert
user
cygerts
67a9f4900b97667e0a82ad40
Bartłomiej Twardowski
67a9f4900b97667e0a82ad41
Andrew D. Bagdanov
67a9f4900b97667e0a82ad42
Joost van de Weijer
No Task Left Behind: Isotropic Model Merging with Common and
  Task-Specific Subspaces
Model merging integrates the weights of multiple task-specific models into a
single multi-task model. Despite recent interest in the problem, a significant
performance gap between the combined and single-task models remains. In this
paper, we investigate the key characteristics of task matrices -- weight update
matrices applied to a pre-trained model -- that enable effective merging. We
show that alignment between singular components of task-specific and merged
matrices strongly correlates with performance improvement over the pre-trained
model. Based on this, we propose an isotropic merging framework that flattens
the singular value spectrum of task matrices, enhances alignment, and reduces
the performance gap. Additionally, we incorporate both common and task-specific
subspaces to further improve alignment and performance. Our proposed approach
achieves state-of-the-art performance across multiple scenarios, including
various sets of tasks and model scales. This work advances the understanding of
model merging dynamics, offering an effective methodology to merge models
without requiring additional training. Code is available at
https://github.com/danielm1405/iso-merging .
The paper proposes isotropic merging to improve performance during model merging by aligning singular components and incorporating common and task-specific subspaces.
task matrices weight update matrices singular components isotropic merging singular value spectrum common subspaces task-specific subspaces