DELLA-Merging: Reducing Interference in Model Merging through
  Magnitude-Based Sampling
With the proliferation of domain-specific models, model merging has emerged
as a set of techniques that combine the capabilities of multiple models into
one that can multitask without the cost of additional training. In this paper,
we propose a new model merging technique, Drop and rEscaLe via sampLing with
mAgnitude (DELLA-Merging), that employs a novel pruning technique, MAGPRUNE,
which shows significant advantages over DARE and TIES. MAGPRUNE first ranks the
parameters in order of their magnitude and assigns higher dropout probabilities
(p) to parameters with lower ranks corresponding to lower magnitudes. To
approximate the original embeddings, MAGPRUNE employs a rescaling operation on
the parameters that survive the random dropping by 1/(1 - p). On three
different expert models considered for merging (LM, Math, Code) and
corresponding benchmark datasets (AlpacaEval, GSM8K, MBPP), DELLA shows an
average improvement of 2.4 points over baseline methods employing delta
parameter pruning (an improvement of 3.6 points over TIES, 1.2 points over
DARE), and 11.1 points over the no-pruning baseline (TA). We release the source
code at: https://github.com/declare-lab/della.
66798918e93f2212af0ddb59
Pala Tej Deep
claimed_verified
2024-06-24T20:18:15.135Z
633f875450d83f5065d459d6
/avatars/48312867ac2c05d589869972f90881b9.svg
Pala Tej Deep
user
Tej3
66798918e93f2212af0ddb5a
Rishabh Bhardwaj
claimed_verified
2024-09-19T07:08:25.477Z
5f278507e923d665e616271b
https://cdn-avatars.huggingface.co/v1/production/uploads/5f278507e923d665e616271b/tWFuswXOTXtvMdL8zSrr_.png
Rishabh Bhardwaj
user
RishabhBhardwaj
66798918e93f2212af0ddb5b
Soujanya Poria
claimed_verified
2024-08-22T19:52:37.753Z
626b626405fe1cb65725aca1
https://cdn-avatars.huggingface.co/v1/production/uploads/626b626405fe1cb65725aca1/ZVSbhynzpQhVGq9kGywW6.png
Soujanya Poria
user
soujanyaporia
DELLA-Merging: Reducing Interference in Model Merging through
  Magnitude-Based Sampling
With the proliferation of domain-specific models, model merging has emerged
as a set of techniques that combine the capabilities of multiple models into
one that can multitask without the cost of additional training. In this paper,
we propose a new model merging technique, Drop and rEscaLe via sampLing with
mAgnitude (DELLA-Merging), that employs a novel pruning technique, MAGPRUNE,
which shows significant advantages over DARE and TIES. MAGPRUNE first ranks the
parameters in order of their magnitude and assigns higher dropout probabilities
(p) to parameters with lower ranks corresponding to lower magnitudes. To
approximate the original embeddings, MAGPRUNE employs a rescaling operation on
the parameters that survive the random dropping by 1/(1 - p). On three
different expert models considered for merging (LM, Math, Code) and
corresponding benchmark datasets (AlpacaEval, GSM8K, MBPP), DELLA shows an
average improvement of 2.4 points over baseline methods employing delta
parameter pruning (an improvement of 3.6 points over TIES, 1.2 points over
DARE), and 11.1 points over the no-pruning baseline (TA). We release the source
code at: https://github.com/declare-lab/della.
A novel model merging technique, DELLA-Merging, using MAGPRUNE pruning significantly enhances multitasking capabilities of domain-specific models across various benchmarks.
model merging DELLA-Merging MAGPRUNE pruning parameter magnitude dropout probabilities rescaling expertise models AlpacaEval GSM8K MBPP delta parameter pruning TIES DARE no-pruning baseline