Observational Scaling Laws and the Predictability of Language Model
  Performance
Understanding how language model performance varies with scale is critical to
benchmark and algorithm development. Scaling laws are one approach to building
this understanding, but the requirement of training models across many
different scales has limited their use. We propose an alternative,
observational approach that bypasses model training and instead builds scaling
laws from ~80 publically available models. Building a single scaling law from
multiple model families is challenging due to large variations in their
training compute efficiencies and capabilities. However, we show that these
variations are consistent with a simple, generalized scaling law where language
model performance is a function of a low-dimensional capability space, and
model families only vary in their efficiency in converting training compute to
capabilities. Using this approach, we show the surprising predictability of
complex scaling phenomena: we show that several emergent phenomena follow a
smooth, sigmoidal behavior and are predictable from small models; we show that
the agent performance of models such as GPT-4 can be precisely predicted from
simpler non-agentic benchmarks; and we show how to predict the impact of
post-training interventions like Chain-of-Thought and Self-Consistency as
language model capabilities continue to improve.
664b64cc5dfaf30f2d2c05cf
Yangjun Ruan
admin_assigned
2024-05-20T15:23:10.344Z
65619949d2e4352d64365606
/avatars/4241b6e901e146a9ff8b5bc7de3f960a.svg
Yangjun Ruan
user
ryoungj
664b64cc5dfaf30f2d2c05d0
Chris J. Maddison
664b64cc5dfaf30f2d2c05d1
Tatsunori Hashimoto
extracted_pending
2024-05-20T14:57:17.446Z
661595d1b3d0b21da55cde7d
/avatars/ba3fa065536518637d21a5c46cee5dd1.svg
Tatsu Hashimoto
user
thashim
Observational Scaling Laws and the Predictability of Language Model
  Performance
Understanding how language model performance varies with scale is critical to
benchmark and algorithm development. Scaling laws are one approach to building
this understanding, but the requirement of training models across many
different scales has limited their use. We propose an alternative,
observational approach that bypasses model training and instead builds scaling
laws from ~80 publically available models. Building a single scaling law from
multiple model families is challenging due to large variations in their
training compute efficiencies and capabilities. However, we show that these
variations are consistent with a simple, generalized scaling law where language
model performance is a function of a low-dimensional capability space, and
model families only vary in their efficiency in converting training compute to
capabilities. Using this approach, we show the surprising predictability of
complex scaling phenomena: we show that several emergent phenomena follow a
smooth, sigmoidal behavior and are predictable from small models; we show that
the agent performance of models such as GPT-4 can be precisely predicted from
simpler non-agentic benchmarks; and we show how to predict the impact of
post-training interventions like Chain-of-Thought and Self-Consistency as
language model capabilities continue to improve.
An observational approach to scaling laws in language models can predict complex phenomena and agent performance using capability space and efficiency in converting training compute to capabilities.
scaling laws language model performance capability space training compute emergent phenomena sigmoidal behavior predictive modeling Chain-of-Thought Self-Consistency