Scaling Laws for Sparsely-Connected Foundation Models
We explore the impact of parameter sparsity on the scaling behavior of
Transformers trained on massive datasets (i.e., "foundation models"), in both
vision and language domains. In this setting, we identify the first scaling law
describing the relationship between weight sparsity, number of non-zero
parameters, and amount of training data, which we validate empirically across
model and data scales; on ViT/JFT-4B and T5/C4. These results allow us to
characterize the "optimal sparsity", the sparsity level which yields the best
performance for a given effective model size and training budget. For a fixed
number of non-zero parameters, we identify that the optimal sparsity increases
with the amount of data used for training. We also extend our study to
different sparsity structures (such as the hardware-friendly n:m pattern) and
strategies (such as starting from a pretrained dense model). Our findings shed
light on the power and limitations of weight sparsity across various parameter
and computational settings, offering both theoretical understanding and
practical implications for leveraging sparsity towards computational efficiency
improvements.
6507b9dfaf387baff4a0a8d3
Elias Frantar
admin_assigned
2023-09-18T09:57:07.223Z
64b79f81037d6452a317b846
/avatars/c9c9770136804f8e47f52ff1a4cd120a.svg
Elias Frantar
user
efrantar
6507b9dfaf387baff4a0a8d4
Carlos Riquelme
6507b9dfaf387baff4a0a8d5
Neil Houlsby
admin_assigned
2023-09-18T09:57:25.476Z
64a2c7f263f69fb98d3bfdb4
/avatars/a362a236c0654b7605dcb7673e309335.svg
Neil Houlsby
user
neilhoulsby
6507b9dfaf387baff4a0a8d6
Dan Alistarh
admin_assigned
2023-09-18T09:57:33.186Z
64d100c5d8d0927372e3d4c0
/avatars/91d9e4f1dab25b70d901783cdfcd2fd1.svg
Dan Alistarh
user
dalistarh
6507b9dfaf387baff4a0a8d7
Utku Evci
claimed_verified
2023-09-18T19:42:48.174Z
65088f3b623330a3a5fcd55a
/avatars/487230672d080d1084fede5e92a3a81f.svg
Evci
user
evcu
Scaling Laws for Sparsely-Connected Foundation Models
We explore the impact of parameter sparsity on the scaling behavior of
Transformers trained on massive datasets (i.e., "foundation models"), in both
vision and language domains. In this setting, we identify the first scaling law
describing the relationship between weight sparsity, number of non-zero
parameters, and amount of training data, which we validate empirically across
model and data scales; on ViT/JFT-4B and T5/C4. These results allow us to
characterize the "optimal sparsity", the sparsity level which yields the best
performance for a given effective model size and training budget. For a fixed
number of non-zero parameters, we identify that the optimal sparsity increases
with the amount of data used for training. We also extend our study to
different sparsity structures (such as the hardware-friendly n:m pattern) and
strategies (such as starting from a pretrained dense model). Our findings shed
light on the power and limitations of weight sparsity across various parameter
and computational settings, offering both theoretical understanding and
practical implications for leveraging sparsity towards computational efficiency
improvements.
Parameter sparsity in Transformers affects their scaling behavior on large datasets; optimal sparsity increases with data volume and different structures can impact performance.
parameter sparsity scaling law Transformers weight sparsity ViT JFT-4B T5 C4 optimal sparsity hardware-friendly n:m pattern pretrained dense model