The Languini Kitchen: Enabling Language Modelling Research at Different
  Scales of Compute
The Languini Kitchen serves as both a research collective and codebase
designed to empower researchers with limited computational resources to
contribute meaningfully to the field of language modelling. We introduce an
experimental protocol that enables model comparisons based on equivalent
compute, measured in accelerator hours. The number of tokens on which a model
is trained is defined by the model's throughput and the chosen compute class.
Notably, this approach avoids constraints on critical hyperparameters which
affect total parameters or floating-point operations. For evaluation, we
pre-process an existing large, diverse, and high-quality dataset of books that
surpasses existing academic benchmarks in quality, diversity, and document
length. On it, we compare methods based on their empirical scaling trends which
are estimated through experiments at various levels of compute. This work also
provides two baseline models: a feed-forward model derived from the GPT-2
architecture and a recurrent model in the form of a novel LSTM with ten-fold
throughput. While the GPT baseline achieves better perplexity throughout all
our levels of compute, our LSTM baseline exhibits a predictable and more
favourable scaling law. This is due to the improved throughput and the need for
fewer training tokens to achieve the same decrease in test perplexity.
Extrapolating the scaling laws leads of both models results in an intersection
at roughly 50,000 accelerator hours. We hope this work can serve as the
foundation for meaningful and reproducible language modelling research.
650bd3f00f4a84a5db565e92
Aleksandar Stanić
650bd3f00f4a84a5db565e93
Dylan Ashley
admin_assigned
2023-09-21T10:02:06.175Z
63ff35c44852102d4873de7d
https://cdn-avatars.huggingface.co/v1/production/uploads/1677669808298-noauth.png
Dylan Ashley
user
dylanashley
650bd3f00f4a84a5db565e94
Oleg Serikov
admin_assigned
2023-09-21T10:02:13.749Z
614d9b7e398ee2c4c1274f2e
/avatars/ace33cf5b77e81eb7702a27ee53c4000.svg
Oleg Serikov
user
oserikov
650bd3f00f4a84a5db565e95
Louis Kirsch
650bd3f00f4a84a5db565e96
Francesco Faccio
650bd3f00f4a84a5db565e97
Jürgen Schmidhuber
650bd3f00f4a84a5db565e98
Thomas Hofmann
650bd3f00f4a84a5db565e99
Imanol Schlag
extracted_pending
2023-09-21T05:26:10.064Z
6310fac907a768279024d158
/avatars/5bced71b6d507e621ec159ada94480f0.svg
Imanol Schlag
user
ischlag
The Languini Kitchen: Enabling Language Modelling Research at Different
  Scales of Compute
The Languini Kitchen serves as both a research collective and codebase
designed to empower researchers with limited computational resources to
contribute meaningfully to the field of language modelling. We introduce an
experimental protocol that enables model comparisons based on equivalent
compute, measured in accelerator hours. The number of tokens on which a model
is trained is defined by the model's throughput and the chosen compute class.
Notably, this approach avoids constraints on critical hyperparameters which
affect total parameters or floating-point operations. For evaluation, we
pre-process an existing large, diverse, and high-quality dataset of books that
surpasses existing academic benchmarks in quality, diversity, and document
length. On it, we compare methods based on their empirical scaling trends which
are estimated through experiments at various levels of compute. This work also
provides two baseline models: a feed-forward model derived from the GPT-2
architecture and a recurrent model in the form of a novel LSTM with ten-fold
throughput. While the GPT baseline achieves better perplexity throughout all
our levels of compute, our LSTM baseline exhibits a predictable and more
favourable scaling law. This is due to the improved throughput and the need for
fewer training tokens to achieve the same decrease in test perplexity.
Extrapolating the scaling laws leads of both models results in an intersection
at roughly 50,000 accelerator hours. We hope this work can serve as the
foundation for meaningful and reproducible language modelling research.
A protocol for comparing language models based on equivalent computational resources evaluates GPT-2 and a new high-throughput LSTM, showing predictable scaling and intersection in performance at 50,000 accelerator hours.
feed-forward model GPT-2 architecture recurrent model LSTM perplexity throughput empirical scaling trends accelerator hours training tokens document length