MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient
  Fine-Tuning of Large Language Models
Recent studies integrate Low-Rank Adaptation (LoRA) and Mixture-of-Experts
(MoE) to further enhance the performance of parameter-efficient fine-tuning
(PEFT) methods in Large Language Model (LLM) applications. Existing methods
employ homogeneous MoE-LoRA architectures composed of LoRA experts with
either similar or identical structures and capacities. However, these
approaches often suffer from representation collapse and expert load imbalance,
which negatively impact the potential of LLMs. To address these challenges, we
propose a heterogeneous Mixture-of-Adapters (MoA) approach.
This method dynamically integrates PEFT adapter experts with diverse
structures, leveraging their complementary representational capabilities to
foster expert specialization, thereby enhancing the effective transfer of
pre-trained knowledge to downstream tasks. MoA supports two variants:
(i) Soft MoA achieves fine-grained integration by performing
a weighted fusion of all expert outputs; (ii) Sparse MoA
activates adapter experts sparsely based on their contribution, achieving this
with negligible performance degradation. Experimental results demonstrate that
heterogeneous MoA outperforms homogeneous MoE-LoRA methods in both performance
and parameter efficiency. Our project is available at
https://github.com/DCDmllm/MoA.
6847b3393ec10bdd8ab4df20
Jie Cao
claimed_verified
2025-06-10T08:42:50.209Z
65ea90741b0d7e029a3a1fb0
/avatars/7a4f861d4ead080996bb28e2b6cb8ac5.svg
cj
user
cajie
6847b3393ec10bdd8ab4df21
Tianwei Lin
6847b3393ec10bdd8ab4df22
Hongyang He
6847b3393ec10bdd8ab4df23
Rolan Yan
6847b3393ec10bdd8ab4df24
Wenqiao Zhang
6847b3393ec10bdd8ab4df25
Juncheng Li
6847b3393ec10bdd8ab4df26
Dongping Zhang
6847b3393ec10bdd8ab4df27
Siliang Tang
6847b3393ec10bdd8ab4df28
Yueting Zhuang
MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient
  Fine-Tuning of Large Language Models
Recent studies integrate Low-Rank Adaptation (LoRA) and Mixture-of-Experts
(MoE) to further enhance the performance of parameter-efficient fine-tuning
(PEFT) methods in Large Language Model (LLM) applications. Existing methods
employ homogeneous MoE-LoRA architectures composed of LoRA experts with
either similar or identical structures and capacities. However, these
approaches often suffer from representation collapse and expert load imbalance,
which negatively impact the potential of LLMs. To address these challenges, we
propose a heterogeneous Mixture-of-Adapters (MoA) approach.
This method dynamically integrates PEFT adapter experts with diverse
structures, leveraging their complementary representational capabilities to
foster expert specialization, thereby enhancing the effective transfer of
pre-trained knowledge to downstream tasks. MoA supports two variants:
(i) Soft MoA achieves fine-grained integration by performing
a weighted fusion of all expert outputs; (ii) Sparse MoA
activates adapter experts sparsely based on their contribution, achieving this
with negligible performance degradation. Experimental results demonstrate that
heterogeneous MoA outperforms homogeneous MoE-LoRA methods in both performance
and parameter efficiency. Our project is available at
https://github.com/DCDmllm/MoA.
A heterogeneous Mixture-of-Adapters (MoA) approach enhances parameter-efficient fine-tuning in LLMs by integrating diverse adapter experts, outperforming homogeneous MoE-LoRA methods.
Low-Rank Adaptation Mixture-of-Experts parameter-efficient fine-tuning Large Language Model homogeneous representation collapse expert load imbalance heterogeneous Mixture-of-Adapters soft MoA sparse MoA