μ-Parametrization for Mixture of Experts
Recent years have seen a growing interest and adoption of LLMs, with
muTransfer becoming a key technique for tuning hyperparameters in
large-scale training. Meanwhile, Mixture-of-Experts (MoE) has emerged as a
leading architecture in extremely large models. However, the intersection of
these two advancements has remained unexplored. In this work, we derive a
mu-Parameterization (muP) for MoE, providing theoretical guarantees for
feature learning across model widths in both the router and experts. We
empirically validate our parameterization and further investigate how scaling
the number of experts and granularity affects the optimal learning rate.
689e56b7a4caabb4320e5ca4
Jan Małaśnicki
689e56b7a4caabb4320e5ca5
Kamil Ciebiera
689e56b7a4caabb4320e5ca6
Mateusz Boruń
689e56b7a4caabb4320e5ca7
Maciej Pióro
689e56b7a4caabb4320e5ca8
Jan Ludziejewski
689e56b7a4caabb4320e5ca9
Maciej Stefaniak
689e56b7a4caabb4320e5caa
Michał Krutul
689e56b7a4caabb4320e5cab
Sebastian Jaszczur
689e56b7a4caabb4320e5cac
Marek Cygan
689e56b7a4caabb4320e5cad
Kamil Adamczewski
689e56b7a4caabb4320e5cae
Jakub Krajewski
μ-Parametrization for Mixture of Experts
Recent years have seen a growing interest and adoption of LLMs, with
muTransfer becoming a key technique for tuning hyperparameters in
large-scale training. Meanwhile, Mixture-of-Experts (MoE) has emerged as a
leading architecture in extremely large models. However, the intersection of
these two advancements has remained unexplored. In this work, we derive a
mu-Parameterization (muP) for MoE, providing theoretical guarantees for
feature learning across model widths in both the router and experts. We
empirically validate our parameterization and further investigate how scaling
the number of experts and granularity affects the optimal learning rate.
A new parameterization for Mixture-of-Experts models provides theoretical guarantees for feature learning and investigates the impact of scaling experts and granularity on learning rates.
Mixture-of-Experts MoE $\mu$Transfer $\mu$-Parameterization $\mu$P router experts feature learning learning rate