Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and
  Uncertainty Based Routing
Model merging aggregates Large Language Models (LLMs) finetuned on different
tasks into a stronger one. However, parameter conflicts between models leads to
performance degradation in averaging. While model routing addresses this issue
by selecting individual models during inference, it imposes excessive storage
and compute costs, and fails to leverage the common knowledge from different
models. In this work, we observe that different layers exhibit varying levels
of parameter conflicts. Building on this insight, we average layers with
minimal parameter conflicts and use a novel task-level expert routing for
layers with significant conflicts. To further reduce storage costs, inspired by
task arithmetic sparsity, we decouple multiple fine-tuned experts into a dense
expert and several sparse experts. Considering the out-of-distribution samples,
we select and merge appropriate experts based on the task uncertainty of the
input data. We conduct extensive experiments on both LLaMA and Qwen with
varying parameter scales, and evaluate on real-world reasoning tasks. Results
demonstrate that our method consistently achieves significant performance
improvements while requiring less system cost compared to existing methods.
67adad972883187d78409a7a
Kunfeng Lai
67adad972883187d78409a7b
Zhenheng Tang
67adad972883187d78409a7c
Xinglin Pan
67adad972883187d78409a7d
Peijie Dong
claimed_verified
2025-06-02T07:53:30.947Z
6395f845aec00abff778ad31
https://cdn-avatars.huggingface.co/v1/production/uploads/6395f845aec00abff778ad31/bZkAlchSvqER1HgBKmcHI.jpeg
PeijieDong
user
pprp
67adad972883187d78409a7e
Xiang Liu
claimed_verified
2025-02-13T08:45:17.030Z
63024676056ec3a2a8714b24
https://cdn-avatars.huggingface.co/v1/production/uploads/1661093436322-noauth.jpeg
Xiang Liu
user
Dominic789654
67adad972883187d78409a7f
Haolan Chen
67adad972883187d78409a80
Li Shen
67adad972883187d78409a81
Bo Li
67adad972883187d78409a82
Xiaowen Chu
Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and
  Uncertainty Based Routing
Model merging aggregates Large Language Models (LLMs) finetuned on different
tasks into a stronger one. However, parameter conflicts between models leads to
performance degradation in averaging. While model routing addresses this issue
by selecting individual models during inference, it imposes excessive storage
and compute costs, and fails to leverage the common knowledge from different
models. In this work, we observe that different layers exhibit varying levels
of parameter conflicts. Building on this insight, we average layers with
minimal parameter conflicts and use a novel task-level expert routing for
layers with significant conflicts. To further reduce storage costs, inspired by
task arithmetic sparsity, we decouple multiple fine-tuned experts into a dense
expert and several sparse experts. Considering the out-of-distribution samples,
we select and merge appropriate experts based on the task uncertainty of the
input data. We conduct extensive experiments on both LLaMA and Qwen with
varying parameter scales, and evaluate on real-world reasoning tasks. Results
demonstrate that our method consistently achieves significant performance
improvements while requiring less system cost compared to existing methods.
A method for merging Large Language Models leverages layer-wise averaging and task-level expert routing to improve performance and reduce system cost.
Model merging Large Language Models LLMs parameter conflicts model routing layer-wise averaging task-level expert routing task arithmetic sparsity dense expert sparse experts out-of-distribution samples task uncertainty LLaMA Qwen