Scaling Laws for Fine-Grained Mixture of Experts
Mixture of Experts (MoE) models have emerged as a primary solution for
reducing the computational cost of Large Language Models. In this work, we
analyze their scaling properties, incorporating an expanded range of variables.
Specifically, we introduce a new hyperparameter, granularity, whose adjustment
enables precise control over the size of the experts. Building on this, we
establish scaling laws for fine-grained MoE, taking into account the number of
training tokens, model size, and granularity. Leveraging these laws, we derive
the optimal training configuration for a given computational budget. Our
findings not only show that MoE models consistently outperform dense
Transformers but also highlight that the efficiency gap between dense and MoE
models widens as we scale up the model size and training budget. Furthermore,
we demonstrate that the common practice of setting the size of experts in MoE
to mirror the feed-forward layer is not optimal at almost any computational
budget.
65cae5a25f6633f8ce245f08
Jakub Krajewski
65cae5a25f6633f8ce245f09
Jan Ludziejewski
65cae5a25f6633f8ce245f0a
Kamil Adamczewski
65cae5a25f6633f8ce245f0b
Maciej Pióro
admin_assigned
2024-02-13T12:03:19.763Z
643e7c8b3870da7f0ef20f36
/avatars/71291f75e2ab20818537159179f904a6.svg
Maciej Pióro
user
maciek-pioro
65cae5a25f6633f8ce245f0c
Michał Krutul
admin_assigned
2024-02-13T12:03:27.911Z
63bf063987619d1458ca9177
/avatars/58f55bb6dd6f23b00fd2f01bd3be543a.svg
Michał Krutul
user
crewtool
65cae5a25f6633f8ce245f0d
Szymon Antoniak
admin_assigned
2024-02-13T12:03:34.919Z
63f767c4bd28622c9b984637
/avatars/126a23ca5b3adc89418b7a53b21e50e6.svg
Szymon Antoniak
user
Simontwice
65cae5a25f6633f8ce245f0e
Kamil Ciebiera
65cae5a25f6633f8ce245f0f
Krystian Król
65cae5a25f6633f8ce245f10
Tomasz Odrzygóźdź
65cae5a25f6633f8ce245f11
Piotr Sankowski
65cae5a25f6633f8ce245f12
Marek Cygan
65cae5a25f6633f8ce245f13
Sebastian Jaszczur
Scaling Laws for Fine-Grained Mixture of Experts
Mixture of Experts (MoE) models have emerged as a primary solution for
reducing the computational cost of Large Language Models. In this work, we
analyze their scaling properties, incorporating an expanded range of variables.
Specifically, we introduce a new hyperparameter, granularity, whose adjustment
enables precise control over the size of the experts. Building on this, we
establish scaling laws for fine-grained MoE, taking into account the number of
training tokens, model size, and granularity. Leveraging these laws, we derive
the optimal training configuration for a given computational budget. Our
findings not only show that MoE models consistently outperform dense
Transformers but also highlight that the efficiency gap between dense and MoE
models widens as we scale up the model size and training budget. Furthermore,
we demonstrate that the common practice of setting the size of experts in MoE
to mirror the feed-forward layer is not optimal at almost any computational
budget.
MoE models reduce computational cost through fine-grained scaling controlled by a hyperparameter called granularity, outperforming dense Transformers across varying scales and budgets.
Mixture of Experts (MoE) scaling properties hyperparameter granularity scaling laws fine-grained MoE training tokens model size optimal training configuration computational budget dense Transformers feed-forward layer