On Linear Mode Connectivity of Mixture-of-Experts Architectures
Hoang Tran, Van Hoan Trinh, VINH BUI, Tan Nguyen
Linear Mode Connectivity (LMC) is a notable phenomenon in the loss landscapesof neural networks, wherein independently trained models have been observed tobe connected—up to permutation symmetries—by linear paths in parameter spacealong which the loss remains consistently low. This observation challenges classicalviews of non-convex optimization and has implications for model ensembling,generalization, and our understanding of neural loss geometry. Inspired by recentstudies on LMC in standard neural networks, we systematically investigate thisphenomenon within Mixture-of-Experts (MoE) architectures—a class of modelsknown for their scalability and computational efficiency, which combine traditionalneural networks—referred to as experts—through a learnable gating mechanism.We begin by conducting a comprehensive analysis of both dense and sparse gatingregimes, demonstrating that the symmetries inherent to MoE architectures arefully characterized by permutations acting on both the expert components and thegating function. Building on these foundational findings, we propose a matchingalgorithm that enables alignment between independently trained MoEs, therebyfacilitating the discovery of LMC. Finally, we empirically validate the presence ofLMC using our proposed algorithm across diverse MoE configurations—includingdense, sparse, and shared-expert variants—under a wide range of model settingsand datasets of varying scales and modalities. Our results confirm the existenceof LMC in MoE architectures and offer fundamental insights into the functionallandscape and optimization dynamics of deep learning models.