Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for
  Sparse Architectural Large Language Models
Parameter-efficient fine-tuning (PEFT) is crucial for customizing Large
Language Models (LLMs) with constrained resources. Although there have been
various PEFT methods for dense-architecture LLMs, PEFT for sparse-architecture
LLMs is still underexplored. In this work, we study the PEFT method for LLMs
with the Mixture-of-Experts (MoE) architecture and the contents of this work
are mainly threefold: (1) We investigate the dispersion degree of the activated
experts in customized tasks, and found that the routing distribution for a
specific task tends to be highly concentrated, while the distribution of
activated experts varies significantly across different tasks. (2) We propose
Expert-Specialized Fine-Tuning, or ESFT, which tunes the experts most relevant
to downstream tasks while freezing the other experts and modules; experimental
results demonstrate that our method not only improves the tuning efficiency,
but also matches or even surpasses the performance of full-parameter
fine-tuning. (3) We further analyze the impact of the MoE architecture on
expert-specialized fine-tuning. We find that MoE models with finer-grained
experts are more advantageous in selecting the combination of experts that are
most relevant to downstream tasks, thereby enhancing both the training
efficiency and effectiveness.
6686701e443833e90fc83875
Zihan Wang
claimed_verified
2025-05-02T06:41:08.760Z
63318e014bd187e9ce05cec6
https://cdn-avatars.huggingface.co/v1/production/uploads/63318e014bd187e9ce05cec6/rKN84BAXF_u0osSu6KJy5.jpeg
Zihan Wang
user
ZihanWang314
6686701e443833e90fc83876
Deli Chen
6686701e443833e90fc83877
Damai Dai
6686701e443833e90fc83878
Runxin Xu
6686701e443833e90fc83879
Zhuoshu Li
6686701e443833e90fc8387a
Y. Wu
Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for
  Sparse Architectural Large Language Models
Parameter-efficient fine-tuning (PEFT) is crucial for customizing Large
Language Models (LLMs) with constrained resources. Although there have been
various PEFT methods for dense-architecture LLMs, PEFT for sparse-architecture
LLMs is still underexplored. In this work, we study the PEFT method for LLMs
with the Mixture-of-Experts (MoE) architecture and the contents of this work
are mainly threefold: (1) We investigate the dispersion degree of the activated
experts in customized tasks, and found that the routing distribution for a
specific task tends to be highly concentrated, while the distribution of
activated experts varies significantly across different tasks. (2) We propose
Expert-Specialized Fine-Tuning, or ESFT, which tunes the experts most relevant
to downstream tasks while freezing the other experts and modules; experimental
results demonstrate that our method not only improves the tuning efficiency,
but also matches or even surpasses the performance of full-parameter
fine-tuning. (3) We further analyze the impact of the MoE architecture on
expert-specialized fine-tuning. We find that MoE models with finer-grained
experts are more advantageous in selecting the combination of experts that are
most relevant to downstream tasks, thereby enhancing both the training
efficiency and effectiveness.
Research on parameter-efficient fine-tuning for Large Language Models with Mixture-of-Experts architecture reveals that task-specific expert tuning can enhance performance and efficiency.
Parameter-efficient fine-tuning PEFT Large Language Models LLMs Mixture-of-Experts MoE dispersion degree activated experts routing distribution Expert-Specialized Fine-Tuning ESFT full-parameter fine-tuning training efficiency effectiveness