Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for
  Mixture-of-Experts Language Models
Scaling the capacity of language models has consistently proven to be a
reliable approach for improving performance and unlocking new capabilities.
Capacity can be primarily defined by two dimensions: the number of model
parameters and the compute per example. While scaling typically involves
increasing both, the precise interplay between these factors and their combined
contribution to overall capacity remains not fully understood. We explore this
relationship in the context of sparse Mixture-of-Experts (MoEs), which allow
scaling the number of parameters without proportionally increasing the FLOPs
per example. We investigate how varying the sparsity level, i.e., the fraction
of inactive parameters, impacts model's performance during pretraining and
downstream few-shot evaluation. We find that under different constraints (e.g.,
parameter size and total training compute), there is an optimal level of
sparsity that improves both training efficiency and model performance. These
results provide a better understanding of the impact of sparsity in scaling
laws for MoEs and complement existing works in this area, offering insights for
designing more efficient architectures.
6798d09d208ffebef5bcfa47
Samira Abnar
6798d09d208ffebef5bcfa48
Harshay Shah
admin_assigned
2025-01-28T14:16:56.095Z
64b1a4f64dd3e24895daa236
https://cdn-avatars.huggingface.co/v1/production/uploads/64b1a4f64dd3e24895daa236/lzlZ4DBOw-YVspNttAEY3.jpeg
Harshay Shah
user
harshay
6798d09d208ffebef5bcfa49
Dan Busbridge
admin_assigned
2025-01-28T14:17:03.153Z
64c3726f2a5eaefd000cdedd
https://cdn-avatars.huggingface.co/v1/production/uploads/64c3726f2a5eaefd000cdedd/iwFifH1sWQy7agW3eTmNQ.png
Dan Busbridge
user
dbusbridge
6798d09d208ffebef5bcfa4a
Alaaeldin Mohamed Elnouby Ali
6798d09d208ffebef5bcfa4b
Josh Susskind
6798d09d208ffebef5bcfa4c
Vimal Thilak
admin_assigned
2025-01-28T14:17:43.503Z
6737e918edaf7e05e4b35791
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/uk--P2mE2Jeg1QoxmXSTS.png
Vimal Thilak
user
vimalthilak
Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for
  Mixture-of-Experts Language Models
Scaling the capacity of language models has consistently proven to be a
reliable approach for improving performance and unlocking new capabilities.
Capacity can be primarily defined by two dimensions: the number of model
parameters and the compute per example. While scaling typically involves
increasing both, the precise interplay between these factors and their combined
contribution to overall capacity remains not fully understood. We explore this
relationship in the context of sparse Mixture-of-Experts (MoEs), which allow
scaling the number of parameters without proportionally increasing the FLOPs
per example. We investigate how varying the sparsity level, i.e., the fraction
of inactive parameters, impacts model's performance during pretraining and
downstream few-shot evaluation. We find that under different constraints (e.g.,
parameter size and total training compute), there is an optimal level of
sparsity that improves both training efficiency and model performance. These
results provide a better understanding of the impact of sparsity in scaling
laws for MoEs and complement existing works in this area, offering insights for
designing more efficient architectures.
Research on sparse Mixture-of-Experts (MoEs) shows an optimal level of sparsity improves both training efficiency and model performance, providing insights for designing efficient architectures.
Mixture-of-Experts sparsity inactive parameters few-shot evaluation scaling laws