Algorithmic progress in language models
We investigate the rate at which algorithms for pre-training language models
have improved since the advent of deep learning. Using a dataset of over 200
language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we
find that the compute required to reach a set performance threshold has halved
approximately every 8 months, with a 95% confidence interval of around 5 to 14
months, substantially faster than hardware gains per Moore's Law. We estimate
augmented scaling laws, which enable us to quantify algorithmic progress and
determine the relative contributions of scaling models versus innovations in
training algorithms. Despite the rapid pace of algorithmic progress and the
development of new architectures such as the transformer, our analysis reveals
that the increase in compute made an even larger contribution to overall
performance improvements over this time period. Though limited by noisy
benchmark data, our analysis quantifies the rapid progress in language
modeling, shedding light on the relative contributions from compute and
algorithms.
65efcbd5f62dffb225ce0e35
Anson Ho
65efcbd5f62dffb225ce0e36
Tamay Besiroglu
65efcbd5f62dffb225ce0e37
Ege Erdil
65efcbd5f62dffb225ce0e38
David Owen
admin_assigned
2024-03-12T12:37:31.164Z
646f5bf8075e11ca78dbb4e5
/avatars/7dea81e7e8f4be92e99ebcfa39150dc2.svg
David Owen
user
davidowen-epoch
65efcbd5f62dffb225ce0e39
Robi Rahman
admin_assigned
2024-03-12T12:37:16.997Z
6552600d1a60bb531b61025e
/avatars/36c7c126244e5f25ae65f7e8562a89c2.svg
Robi Rahman
user
robi-epoch
65efcbd5f62dffb225ce0e3a
Zifan Carl Guo
admin_assigned
2024-03-12T12:37:41.899Z
6333d2ff518c0f78a6e6f8e2
/avatars/b67cda187d941341794c9da3c7dc3b39.svg
Zifan Guo
user
luminolblue
65efcbd5f62dffb225ce0e3b
David Atkinson
claimed_verified
2024-04-12T14:48:57.192Z
64769e3c3d8d1cb791909f09
/avatars/c09f6f1df52a710db6a65e37a4545458.svg
David Atkinson
user
datkinson
65efcbd5f62dffb225ce0e3c
Neil Thompson
admin_assigned
2024-03-12T12:38:13.562Z
636ec54587545ca5a13a4997
/avatars/d3d7ba9515d0d7c02a92e6e914dea1ec.svg
Neil Thompson
user
neilqvt
65efcbd5f62dffb225ce0e3d
Jaime Sevilla
admin_assigned
2024-03-12T12:38:19.528Z
60d2e60d07da9c17c72708f9
https://cdn-avatars.huggingface.co/v1/production/uploads/1624434395907-60d2e60d07da9c17c72708f9.png
Jaime Sevilla
user
Jsevillamol
Algorithmic progress in language models
We investigate the rate at which algorithms for pre-training language models
have improved since the advent of deep learning. Using a dataset of over 200
language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we
find that the compute required to reach a set performance threshold has halved
approximately every 8 months, with a 95% confidence interval of around 5 to 14
months, substantially faster than hardware gains per Moore's Law. We estimate
augmented scaling laws, which enable us to quantify algorithmic progress and
determine the relative contributions of scaling models versus innovations in
training algorithms. Despite the rapid pace of algorithmic progress and the
development of new architectures such as the transformer, our analysis reveals
that the increase in compute made an even larger contribution to overall
performance improvements over this time period. Though limited by noisy
benchmark data, our analysis quantifies the rapid progress in language
modeling, shedding light on the relative contributions from compute and
algorithms.
Analysis of language model evaluations shows that compute requirements halve faster than hardware improvements, with compute making the larger contribution to performance gains.
pre-training language models deep learning dataset Wikitext Penn Treebank compute performance threshold Moore's Law augmented scaling laws training algorithms transformer language modeling