Accurate and Efficient Low-Rank Model Merging in Core Space
In this paper, we address the challenges associated with merging low-rank
adaptations of large neural networks. With the rise of parameter-efficient
adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning
has become more accessible. While fine-tuning models with LoRA is highly
efficient, existing merging methods often sacrifice this efficiency by merging
fully-sized weight matrices. We propose the Core Space merging framework, which
enables the merging of LoRA-adapted models within a common alignment basis,
thereby preserving the efficiency of low-rank adaptation while substantially
improving accuracy across tasks. We further provide a formal proof that
projection into Core Space ensures no loss of information and provide a
complexity analysis showing the efficiency gains. Extensive empirical results
demonstrate that Core Space significantly improves existing merging techniques
and achieves state-of-the-art results on both vision and language tasks while
utilizing a fraction of the computational resources. Codebase is available at
https://github.com/apanariello4/core-space-merging.
68d2db0d0e215259d193b11b
Aniello Panariello
claimed_verified
2025-09-24T13:53:45.230Z
651e940a081a74588de506c0
/avatars/ae34ee47818135695c0edf01b0b1508c.svg
Aniello Panariello
user
nellopan
68d2db0d0e215259d193b11c
Daniel Marczak
claimed_verified
2025-09-24T13:54:11.809Z
65a5358ddb5c00652ef24c8d
/avatars/d50b6297584c9b4c2ccd93e64477b940.svg
Daniel Marczak
user
danielm1405
68d2db0d0e215259d193b11d
Simone Magistri
68d2db0d0e215259d193b11e
Angelo Porrello
68d2db0d0e215259d193b11f
Bartłomiej Twardowski
68d2db0d0e215259d193b120
Andrew D. Bagdanov
68d2db0d0e215259d193b121
Simone Calderara
68d2db0d0e215259d193b122
Joost van de Weijer
Accurate and Efficient Low-Rank Model Merging in Core Space
In this paper, we address the challenges associated with merging low-rank
adaptations of large neural networks. With the rise of parameter-efficient
adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning
has become more accessible. While fine-tuning models with LoRA is highly
efficient, existing merging methods often sacrifice this efficiency by merging
fully-sized weight matrices. We propose the Core Space merging framework, which
enables the merging of LoRA-adapted models within a common alignment basis,
thereby preserving the efficiency of low-rank adaptation while substantially
improving accuracy across tasks. We further provide a formal proof that
projection into Core Space ensures no loss of information and provide a
complexity analysis showing the efficiency gains. Extensive empirical results
demonstrate that Core Space significantly improves existing merging techniques
and achieves state-of-the-art results on both vision and language tasks while
utilizing a fraction of the computational resources. Codebase is available at
https://github.com/apanariello4/core-space-merging.
Core Space merging framework improves the accuracy and efficiency of merging low-rank adapted models across tasks without significant computational cost.
Low-Rank Adaptation LoRA Core Space parameter-efficient adaptation merging methods weight matrices formal proof complexity analysis state-of-the-art results vision tasks language tasks