Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with
  Architecture-Routed Mixture-of-Experts
Weight-sharing supernet has become a vital component for performance
estimation in the state-of-the-art (SOTA) neural architecture search (NAS)
frameworks. Although supernet can directly generate different subnetworks
without retraining, there is no guarantee for the quality of these subnetworks
because of weight sharing. In NLP tasks such as machine translation and
pre-trained language modeling, we observe that given the same model
architecture, there is a large performance gap between supernet and training
from scratch. Hence, supernet cannot be directly used and retraining is
necessary after finding the optimal architectures.
  In this work, we propose mixture-of-supernets, a generalized supernet
formulation where mixture-of-experts (MoE) is adopted to enhance the expressive
power of the supernet model, with negligible training overhead. In this way,
different subnetworks do not share the model weights directly, but through an
architecture-based routing mechanism. As a result, model weights of different
subnetworks are customized towards their specific architectures and the weight
generation is learned by gradient descent. Compared to existing weight-sharing
supernet for NLP, our method can minimize the retraining time, greatly
improving training efficiency. In addition, the proposed method achieves the
SOTA performance in NAS for building fast machine translation models, yielding
better latency-BLEU tradeoff compared to HAT, state-of-the-art NAS for MT. We
also achieve the SOTA performance in NAS for building memory-efficient
task-agnostic BERT models, outperforming NAS-BERT and AutoDistil in various
model sizes.
64828d90709d2cdcac47bdc8
Ganesh Jawahar
claimed_verified
2023-06-09T17:06:17.883Z
64835bc7c826244a1fddd0a4
/avatars/20ca0523fa980da7063a0b83e20ef7b2.svg
Ganesh Jawahar
user
ganeshjwhr
64828d90709d2cdcac47bdc9
Haichuan Yang
64828d90709d2cdcac47bdca
Yunyang Xiong
64828d90709d2cdcac47bdcb
Zechun Liu
admin_assigned
2023-06-12T13:42:14.734Z
620ae55fe190de4b8f27ba9d
/avatars/aaa2d0eadbcf96c2eb9059e3d73c2760.svg
Liu
user
Zechun
64828d90709d2cdcac47bdcc
Dilin Wang
64828d90709d2cdcac47bdcd
Fei Sun
64828d90709d2cdcac47bdce
Meng Li
claimed_verified
2025-04-10T07:57:51.809Z
65211f76c709aaca9a8bc5c7
/avatars/4fa7619cb18b7ad6869e8fe1547c02cf.svg
Meng Li
user
ml-pku
64828d90709d2cdcac47bdcf
Aasish Pappu
64828d90709d2cdcac47bdd0
Barlas Oguz
admin_assigned
2023-06-12T13:44:23.294Z
6317cb5b212fce5a3cd97227
/avatars/c2b4eefa7c17cdf2299c07466c361ab0.svg
Barlas Oguz
user
barlaso
64828d90709d2cdcac47bdd1
Muhammad Abdul-Mageed
admin_assigned
2023-06-12T13:44:43.727Z
5feeae8dd8e0b2d24ed258cc
/avatars/1d66840db4ed6dc12bdee7ba8168ea9f.svg
Muhammad Abdul-Mageed
user
mageed
64828d90709d2cdcac47bdd2
Laks V. S. Lakshmanan
64828d90709d2cdcac47bdd3
Raghuraman Krishnamoorthi
64828d90709d2cdcac47bdd4
Vikas Chandra
Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with
  Architecture-Routed Mixture-of-Experts
Weight-sharing supernet has become a vital component for performance
estimation in the state-of-the-art (SOTA) neural architecture search (NAS)
frameworks. Although supernet can directly generate different subnetworks
without retraining, there is no guarantee for the quality of these subnetworks
because of weight sharing. In NLP tasks such as machine translation and
pre-trained language modeling, we observe that given the same model
architecture, there is a large performance gap between supernet and training
from scratch. Hence, supernet cannot be directly used and retraining is
necessary after finding the optimal architectures.
  In this work, we propose mixture-of-supernets, a generalized supernet
formulation where mixture-of-experts (MoE) is adopted to enhance the expressive
power of the supernet model, with negligible training overhead. In this way,
different subnetworks do not share the model weights directly, but through an
architecture-based routing mechanism. As a result, model weights of different
subnetworks are customized towards their specific architectures and the weight
generation is learned by gradient descent. Compared to existing weight-sharing
supernet for NLP, our method can minimize the retraining time, greatly
improving training efficiency. In addition, the proposed method achieves the
SOTA performance in NAS for building fast machine translation models, yielding
better latency-BLEU tradeoff compared to HAT, state-of-the-art NAS for MT. We
also achieve the SOTA performance in NAS for building memory-efficient
task-agnostic BERT models, outperforming NAS-BERT and AutoDistil in various
model sizes.
Mixture-of-supernets with an architecture-based routing mechanism improves NAS efficiency and performance in NLP tasks by minimizing retraining time.
mixture-of-supernets mixture-of-experts (MoE) architecture-based routing mechanism neural architecture search (NAS) machine translation pre-trained language modeling weight-sharing supernet gradient descent latency-BLEU tradeoff BERT models NAS-BERT AutoDistil