Language models scale reliably with over-training and on downstream
  tasks
Scaling laws are useful guides for developing language models, but there are
still gaps between current scaling studies and how language models are
ultimately trained and evaluated. For instance, scaling is usually studied in
the compute-optimal training regime (i.e., "Chinchilla optimal" regime);
however, in practice, models are often over-trained to reduce inference costs.
Moreover, scaling laws mostly predict loss on next-token prediction, but
ultimately models are compared based on downstream task performance. In this
paper, we address both shortcomings. To do so, we create a testbed of 104
models with 0.011B to 6.9B parameters trained with various numbers of tokens on
three data distributions. First, we investigate scaling in the over-trained
regime. We fit scaling laws that extrapolate in both the number of model
parameters and the ratio of training tokens to parameters. This enables us to
predict the validation loss of a 1.4B parameter, 900B token run (i.e.,
32times over-trained) and a 6.9B parameter, 138B token
runx2014each from experiments that take 300times less compute.
Second, we relate the perplexity of a language model to its downstream task
performance via a power law. We use this law to predict top-1 error averaged
over downstream tasks for the two aforementioned models using experiments that
take 20times less compute. Our experiments are available at
https://github.com/mlfoundations/scaling.
65f27788f5cf26fe06294cd4
Samir Yitzhak Gadre
admin_assigned
2024-03-14T08:15:04.958Z
62be1cd77c0388414f012f23
https://cdn-avatars.huggingface.co/v1/production/uploads/1656626356583-noauth.jpeg
Samir Gadre
user
samirg
65f27788f5cf26fe06294cd5
Georgios Smyrnis
65f27788f5cf26fe06294cd6
Vaishaal Shankar
admin_assigned
2024-03-14T08:16:18.510Z
63118add64939fabc0108b28
/avatars/0dfbd5d7f1111ec8798e3a9bcb5f7e77.svg
vaishaal shankar
user
vaishaal
65f27788f5cf26fe06294cd7
Suchin Gururangan
admin_assigned
2024-03-14T08:16:25.673Z
5e9272254957053f60648a01
https://cdn-avatars.huggingface.co/v1/production/uploads/1586657211685-5e9272254957053f60648a01.png
Suchin Gururangan
user
suching
65f27788f5cf26fe06294cd8
Mitchell Wortsman
admin_assigned
2024-03-14T08:17:35.484Z
638e42f96251c8bd7ab84089
/avatars/cb7a7f902846bd9710e2c48a50ce3527.svg
Mitchell Wortsman
user
mitchellw
65f27788f5cf26fe06294cd9
Rulin Shao
admin_assigned
2024-03-14T08:17:42.068Z
6334a0bd31a2be3938c59537
https://cdn-avatars.huggingface.co/v1/production/uploads/6334a0bd31a2be3938c59537/kSetFUWAmJbPQ1KSlNKBr.jpeg
Rulin Shao
user
rulins
65f27788f5cf26fe06294cda
Jean Mercat
admin_assigned
2024-03-14T08:17:49.217Z
62be399f01dc22b4d22fc990
/avatars/3e1f3ed30773cbb410b8a9d3c854cac7.svg
Jean Mercat
user
jmercat
65f27788f5cf26fe06294cdb
Alex Fang
65f27788f5cf26fe06294cdc
Jeffrey Li
65f27788f5cf26fe06294cdd
Sedrick Keh
admin_assigned
2024-03-14T08:18:21.055Z
6365c1158b6bd0d9f6605d3d
/avatars/1e3e6573e0cb2bb1b9d8043045120ee6.svg
Sedrick Keh
user
sedrickkeh
65f27788f5cf26fe06294cde
Rui Xin
65f27788f5cf26fe06294cdf
Marianna Nezhurina
admin_assigned
2024-03-14T08:22:36.319Z
62500f361684e0335e527bc6
/avatars/8d2bc4c9cfa8ea82049196431cc3ebea.svg
Marianna Nezhurina
user
marianna13
65f27788f5cf26fe06294ce0
Igor Vasiljevic
admin_assigned
2024-03-14T08:22:57.245Z
633b2d317af633cbcd03a53e
https://cdn-avatars.huggingface.co/v1/production/uploads/633b2d317af633cbcd03a53e/awtbCqUE4qBusxuu17iFg.png
Igor Vasiljevic
user
ivas-tri
65f27788f5cf26fe06294ce1
Jenia Jitsev
admin_assigned
2024-03-14T08:23:08.231Z
6355b485b8b79340d4630dd5
https://cdn-avatars.huggingface.co/v1/production/uploads/6355b485b8b79340d4630dd5/HIZO4ybweRy48VdCtk2MB.jpeg
Jenia Jitsev
user
JJitsev
65f27788f5cf26fe06294ce2
Alexandros G. Dimakis
65f27788f5cf26fe06294ce3
Gabriel Ilharco
admin_assigned
2024-03-14T08:23:21.671Z
635c1b2986f52b24c1e9fde3
/avatars/d53dee91892cccfd0c4d7353ffb67cbf.svg
Gabriel Ilharco
user
gabrielilharco
65f27788f5cf26fe06294ce4
Shuran Song
65f27788f5cf26fe06294ce5
Thomas Kollar
admin_assigned
2024-03-14T08:23:53.434Z
64358e662d0ed796668cb438
/avatars/6f86547d8691e061e289dd906deedfdb.svg
Thomas Kollar
user
thomas-kollar
65f27788f5cf26fe06294ce6
Yair Carmon
admin_assigned
2024-03-14T08:24:48.066Z
63fe1bd38b3c5087ff85e472
/avatars/c481dab1044436db48c591c20441cd66.svg
Yair Carmon
user
ycarmon
65f27788f5cf26fe06294ce7
Achal Dave
admin_assigned
2024-03-14T08:25:03.657Z
6488e29b78570cedad71eb2d
/avatars/0d8bb52c1e2be8d6aa7c732cf92fc5a9.svg
Achal Dave
user
achal-tri
65f27788f5cf26fe06294ce8
Reinhard Heckel
admin_assigned
2024-03-14T08:25:11.497Z
64385800c04b3b996ea3eb3b
/avatars/0bb710434cdb9822f861917a42c93414.svg
Reinhard Heckel
user
reinhardh
65f27788f5cf26fe06294ce9
Niklas Muennighoff
admin_assigned
2024-03-14T08:25:20.035Z
5f1eb362eec0ad2a071ad6e2
https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
Niklas Muennighoff
user
Muennighoff
65f27788f5cf26fe06294cea
Ludwig Schmidt
admin_assigned
2024-03-14T08:25:44.496Z
6252375c818a5dc29ab9d945
/avatars/cd4f0b220ea36e13ff71157cffe9d9d1.svg
Ludwig Schmidt
user
ludwigschmidt
Language models scale reliably with over-training and on downstream
  tasks
Scaling laws are useful guides for developing language models, but there are
still gaps between current scaling studies and how language models are
ultimately trained and evaluated. For instance, scaling is usually studied in
the compute-optimal training regime (i.e., "Chinchilla optimal" regime);
however, in practice, models are often over-trained to reduce inference costs.
Moreover, scaling laws mostly predict loss on next-token prediction, but
ultimately models are compared based on downstream task performance. In this
paper, we address both shortcomings. To do so, we create a testbed of 104
models with 0.011B to 6.9B parameters trained with various numbers of tokens on
three data distributions. First, we investigate scaling in the over-trained
regime. We fit scaling laws that extrapolate in both the number of model
parameters and the ratio of training tokens to parameters. This enables us to
predict the validation loss of a 1.4B parameter, 900B token run (i.e.,
32times over-trained) and a 6.9B parameter, 138B token
runx2014each from experiments that take 300times less compute.
Second, we relate the perplexity of a language model to its downstream task
performance via a power law. We use this law to predict top-1 error averaged
over downstream tasks for the two aforementioned models using experiments that
take 20times less compute. Our experiments are available at
https://github.com/mlfoundations/scaling.
The study investigates scaling laws in over-trained language models and relates model perplexity to downstream task performance, providing predictions with reduced computational cost.
scaling laws compute-optimal training over-trained regime training tokens validation loss perplexity downstream task performance