Union of Experts: Adapting Hierarchical Routing to Equivalently
  Decomposed Transformer
Mixture-of-Experts (MoE) enhances model performance while maintaining
computational efficiency, making it well-suited for large-scale applications.
However, expert in exist MoE paradigm works as an individual, thereby lacking
high-quality expert interactions. Moreover, they have not been effectively
extended to attention block, which constrains further efficiency improvements.
To tackle these issues, we propose Union-of-Experts (UoE), which decomposes
transformer into an equitant group of experts, and then implement dynamic
routing on input data and experts. Our approach advances MoE design with three
key innovations: (1) We conducted equitant expert decomposition on both MLP
blocks and attention blocks based on matrix partition in tensor parallelism.
(2) We developed two routing paradigms: patch wise data selection and expert
selection, to apply routing across different levels. (3) We design the
architecture of UoE model, including Selective Multi-Head Attention (SMHA) and
Union-of-MLP-Experts (UoME). (4) We develop parallel implementation of UoE's
routing and computation operation, and optimize efficiency based on the
hardware processing analysis. The experiments demonstrate that the model
employed with UoE surpass Full Attention, state-of-art MoEs and efficient
transformers in several tasks across image and natural language domains. The
source codes are available at https://github.com/YujiaoYang-work/UoE.
67c9e3b7668d5bddb4d36a46
Yujiao Yang
claimed_verified
2025-03-07T09:10:00.268Z
67a480dadba32bb665f5aeae
/avatars/ee8d9481a9e86f5e9803bd934a095d61.svg
Yujiao Yang
user
yjyangwork
67c9e3b7668d5bddb4d36a47
Jing Lian
67c9e3b7668d5bddb4d36a48
Linhui Li
Union of Experts: Adapting Hierarchical Routing to Equivalently
  Decomposed Transformer
Mixture-of-Experts (MoE) enhances model performance while maintaining
computational efficiency, making it well-suited for large-scale applications.
However, expert in exist MoE paradigm works as an individual, thereby lacking
high-quality expert interactions. Moreover, they have not been effectively
extended to attention block, which constrains further efficiency improvements.
To tackle these issues, we propose Union-of-Experts (UoE), which decomposes
transformer into an equitant group of experts, and then implement dynamic
routing on input data and experts. Our approach advances MoE design with three
key innovations: (1) We conducted equitant expert decomposition on both MLP
blocks and attention blocks based on matrix partition in tensor parallelism.
(2) We developed two routing paradigms: patch wise data selection and expert
selection, to apply routing across different levels. (3) We design the
architecture of UoE model, including Selective Multi-Head Attention (SMHA) and
Union-of-MLP-Experts (UoME). (4) We develop parallel implementation of UoE's
routing and computation operation, and optimize efficiency based on the
hardware processing analysis. The experiments demonstrate that the model
employed with UoE surpass Full Attention, state-of-art MoEs and efficient
transformers in several tasks across image and natural language domains. The
source codes are available at https://github.com/YujiaoYang-work/UoE.
Union-of-Experts (UoE) improves mixture-of-experts (MoE) by decomposing transformers into groups of experts with dynamic routing, enhancing performance and efficiency in attention and MLP blocks.
Mixture-of-Experts Union-of-Experts transformer equitant expert decomposition MLP blocks attention blocks matrix partition tensor parallelism routing paradigms patch wise data selection expert selection Selective Multi-Head Attention Union-of-MLP-Experts parallel implementation hardware processing analysis Full Attention state-of-the-art MoEs efficient transformers