DivMerge: A divergence-based model merging method for multi-tasking
Multi-task learning (MTL) is often achieved by merging datasets before
fine-tuning, but the growing availability of fine-tuned models has led to new
approaches such as model merging via task arithmetic. A major challenge in this
setting is task interference, which worsens as the number of tasks increases.
We propose a method that merges models trained on different tasks into a single
model, maintaining strong performance across all tasks. Our approach leverages
Jensen-Shannon divergence to guide the merging process without requiring
additional labelled data, and automatically balances task importance. Unlike
existing methods, our approach remains robust as the number of tasks grows and
consistently outperforms prior work.
68c04e3456a3252309f75ce7
Touayouch Brahim
68c04e3456a3252309f75ce8
Fosse Loïc
claimed_verified
2025-09-12T16:19:13.073Z
654b5c8459ff1ee699a0e3eb
/avatars/5c298e978b54f52f41c6c1c4b50f5b7c.svg
loic fosse
user
loicfosse65
68c04e3456a3252309f75ce9
Damnati Géraldine
68c04e3456a3252309f75cea
Lecorvé Gwénolé
DivMerge: A divergence-based model merging method for multi-tasking
Multi-task learning (MTL) is often achieved by merging datasets before
fine-tuning, but the growing availability of fine-tuned models has led to new
approaches such as model merging via task arithmetic. A major challenge in this
setting is task interference, which worsens as the number of tasks increases.
We propose a method that merges models trained on different tasks into a single
model, maintaining strong performance across all tasks. Our approach leverages
Jensen-Shannon divergence to guide the merging process without requiring
additional labelled data, and automatically balances task importance. Unlike
existing methods, our approach remains robust as the number of tasks grows and
consistently outperforms prior work.