Beyond Chinchilla-Optimal: Accounting for Inference in Language Model
  Scaling Laws
Large language model (LLM) scaling laws are empirical formulas that estimate
changes in model quality as a result of increasing parameter count and training
data. However, these formulas, including the popular DeepMind Chinchilla
scaling laws, neglect to include the cost of inference. We modify the
Chinchilla scaling laws to calculate the optimal LLM parameter count and
pre-training data size to train and deploy a model of a given quality and
inference demand. We conduct our analysis both in terms of a compute budget and
real-world costs and find that LLM researchers expecting reasonably large
inference demand (~1B requests) should train models smaller and longer than
Chinchilla-optimal.
65938e54dbdeb5bf078f5644
Nikhil Sardana
extracted_infirmed
2024-01-02T22:38:06.642Z
65938e54dbdeb5bf078f5645
Jonathan Frankle
extracted_pending
2024-01-02T04:17:25.214Z
62b0b7f9a596c50b1783fe9e
https://cdn-avatars.huggingface.co/v1/production/uploads/62b0b7f9a596c50b1783fe9e/nzctJXoyiEwCnOHVJEx3_.png
Jonathan Frankle
user
jfrankle
Beyond Chinchilla-Optimal: Accounting for Inference in Language Model
  Scaling Laws
Large language model (LLM) scaling laws are empirical formulas that estimate
changes in model quality as a result of increasing parameter count and training
data. However, these formulas, including the popular DeepMind Chinchilla
scaling laws, neglect to include the cost of inference. We modify the
Chinchilla scaling laws to calculate the optimal LLM parameter count and
pre-training data size to train and deploy a model of a given quality and
inference demand. We conduct our analysis both in terms of a compute budget and
real-world costs and find that LLM researchers expecting reasonably large
inference demand (~1B requests) should train models smaller and longer than
Chinchilla-optimal.
The modified Chinchilla scaling laws incorporate inference cost, suggesting that models for large inference demand should be smaller and trained longer than originally proposed.
Large language model scaling laws DeepMind Chinchilla parameter count pre-training data inference demand compute budget real-world costs Chinchilla-optimal