Gemstones: A Model Suite for Multi-Faceted Scaling Laws
Scaling laws are typically fit using a family of models with a narrow range
of frozen hyper-parameter choices. In this work we study scaling laws using a
wide range of architecture and hyper-parameter choices, and highlight their
impact on resulting prescriptions. As a primary artifact of our research, we
release the Gemstones: the most comprehensive open-source scaling law dataset
to date, consisting of over 4000 checkpoints from transformers with up to 2
billion parameters; these models have been trained with different learning
rates, cooldown schedules, and architectural shapes. Our checkpoints enable
more complex studies of scaling, such as a law that predicts language modeling
performance as a function of model width and depth. By examining the various
facets of our model suite, we find that the prescriptions of scaling laws can
be highly sensitive to the experimental design process and the specific model
checkpoints used during fitting. Code:
https://github.com/mcleish7/gemstone-scaling-laws
67ac9127184de583cc7daa75
Sean McLeish
extracted_confirmed
2025-02-12T12:21:11.370Z
65255f1073a043e50d043641
/avatars/257085f01c439d7c84787a4e6d085b3d.svg
Sean McLeish
user
smcleish
67ac9127184de583cc7daa76
John Kirchenbauer
67ac9127184de583cc7daa77
David Yu Miller
claimed_verified
2025-02-13T20:35:58.819Z
67ae2de52f89d658be916ff0
/avatars/55f6bea7b6727f5f3f152cd8659f75f6.svg
David Miller
user
dymil
67ac9127184de583cc7daa78
Siddharth Singh
67ac9127184de583cc7daa79
Abhinav Bhatele
67ac9127184de583cc7daa7a
Micah Goldblum
67ac9127184de583cc7daa7b
Ashwinee Panda
67ac9127184de583cc7daa7c
Tom Goldstein
Gemstones: A Model Suite for Multi-Faceted Scaling Laws
Scaling laws are typically fit using a family of models with a narrow range
of frozen hyper-parameter choices. In this work we study scaling laws using a
wide range of architecture and hyper-parameter choices, and highlight their
impact on resulting prescriptions. As a primary artifact of our research, we
release the Gemstones: the most comprehensive open-source scaling law dataset
to date, consisting of over 4000 checkpoints from transformers with up to 2
billion parameters; these models have been trained with different learning
rates, cooldown schedules, and architectural shapes. Our checkpoints enable
more complex studies of scaling, such as a law that predicts language modeling
performance as a function of model width and depth. By examining the various
facets of our model suite, we find that the prescriptions of scaling laws can
be highly sensitive to the experimental design process and the specific model
checkpoints used during fitting. Code:
https://github.com/mcleish7/gemstone-scaling-laws
The study analyzes scaling laws with diverse hyperparameters and architectures, releasing an extensive dataset named Gemstones to facilitate complex scaling law predictions in transformer models.
scaling laws hyper-parameter choices architecture prescription Gemstones language modeling model width model depth