Predicting Task Performance with Context-aware Scaling Laws
Scaling laws have transformed our understanding of large language models by
linking upstream metrics like cross-entropy loss to design factors such as
model size, training data, and compute. However, these conventional laws fail
to capture downstream task performance, where context plays a critical role. In
this work, we propose a straightforward, interpretable framework that jointly
models downstream performance as a function of the training compute and the
provided context. We empirically validate our framework by fitting it on the
observed downstream performance of extended-context variants of Llama-2-7B and
Llama-2-13B across 65,500 unique instances spanning three tasks: arithmetic
reasoning, common sense reasoning, and machine translation. Our results
demonstrate that our framework accurately models in-distribution downstream
performance, generalizes across three orders of magnitude in training compute,
and reliably extrapolates performance as the amount of context increases. These
findings offer valuable insights into the interplay between training compute
and context utilization, providing guidance for designing more efficient
long-context LLMs for diverse downstream tasks. Our code is available at
https://github.com/wang-research-lab/context-scaling.
68f1a9626e0bef323a68fd83
Kyle Montgomery
claimed_verified
2025-10-17T04:10:18.896Z
646c351a31968a60a0237c71
/avatars/d1b3ef3c33671d1cb097bad69e97943b.svg
Kyle Montgomery
user
kylemontgomery
68f1a9626e0bef323a68fd84
David Park
68f1a9626e0bef323a68fd85
Jianhong Tu
68f1a9626e0bef323a68fd86
Michael Bendersky
68f1a9626e0bef323a68fd87
Beliz Gunel
68f1a9626e0bef323a68fd88
Dawn Song
68f1a9626e0bef323a68fd89
Chenguang Wang
Predicting Task Performance with Context-aware Scaling Laws
Scaling laws have transformed our understanding of large language models by
linking upstream metrics like cross-entropy loss to design factors such as
model size, training data, and compute. However, these conventional laws fail
to capture downstream task performance, where context plays a critical role. In
this work, we propose a straightforward, interpretable framework that jointly
models downstream performance as a function of the training compute and the
provided context. We empirically validate our framework by fitting it on the
observed downstream performance of extended-context variants of Llama-2-7B and
Llama-2-13B across 65,500 unique instances spanning three tasks: arithmetic
reasoning, common sense reasoning, and machine translation. Our results
demonstrate that our framework accurately models in-distribution downstream
performance, generalizes across three orders of magnitude in training compute,
and reliably extrapolates performance as the amount of context increases. These
findings offer valuable insights into the interplay between training compute
and context utilization, providing guidance for designing more efficient
long-context LLMs for diverse downstream tasks. Our code is available at
https://github.com/wang-research-lab/context-scaling.
A framework models downstream performance of large language models as a function of training compute and context, offering insights into efficient design for long-context tasks.
scaling laws large language models cross-entropy loss model size training data compute downstream performance context extended-context variants Llama-2-7B Llama-2-13B arithmetic reasoning common sense reasoning machine translation in-distribution performance context utilization long-context LLMs