Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
Scaling the number of parameters and the size of training data has proven to
be an effective strategy for improving large language model (LLM) performance.
Yet, as these models grow increasingly powerful and widely deployed, the cost
of inference has become a pressing concern. Despite its importance, the
trade-off between model accuracy and inference efficiency remains
underexplored. In this work, we examine how key architectural factors, hidden
size, the allocation of parameters between MLP and attention (mlp-to-attention
ratio), and grouped-query attention (GQA), influence both inference cost and
accuracy. We introduce a conditional scaling law that augments the Chinchilla
framework with architectural information, along with a search framework for
identifying architectures that are simultaneously inference-efficient and
accurate. To validate our approach, we train more than 200 models spanning 80M
to 3B parameters and 8B to 100B training tokens, and fit the proposed
conditional scaling law. Our results show that the conditional scaling law
reliably predicts optimal architectural choices and that the resulting models
outperform existing open-source baselines. Under the same training budget,
optimized architectures achieve up to 2.1% higher accuracy and 42% greater
inference throughput compared to LLaMA-3.2.
68f9bf9db9b2e4ae0467393b
Song Bian
claimed_verified
2025-10-23T14:46:06.549Z
620ec8059291e41cab585a3d
https://cdn-avatars.huggingface.co/v1/production/uploads/620ec8059291e41cab585a3d/iuVrjdeghqo-lz0lRfw7l.jpeg
Song
user
NaiveUser
68f9bf9db9b2e4ae0467393c
Tao Yu
68f9bf9db9b2e4ae0467393d
Shivaram Venkataraman
68f9bf9db9b2e4ae0467393e
Youngsuk Park
Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
Scaling the number of parameters and the size of training data has proven to
be an effective strategy for improving large language model (LLM) performance.
Yet, as these models grow increasingly powerful and widely deployed, the cost
of inference has become a pressing concern. Despite its importance, the
trade-off between model accuracy and inference efficiency remains
underexplored. In this work, we examine how key architectural factors, hidden
size, the allocation of parameters between MLP and attention (mlp-to-attention
ratio), and grouped-query attention (GQA), influence both inference cost and
accuracy. We introduce a conditional scaling law that augments the Chinchilla
framework with architectural information, along with a search framework for
identifying architectures that are simultaneously inference-efficient and
accurate. To validate our approach, we train more than 200 models spanning 80M
to 3B parameters and 8B to 100B training tokens, and fit the proposed
conditional scaling law. Our results show that the conditional scaling law
reliably predicts optimal architectural choices and that the resulting models
outperform existing open-source baselines. Under the same training budget,
optimized architectures achieve up to 2.1% higher accuracy and 42% greater
inference throughput compared to LLaMA-3.2.
A conditional scaling law is introduced to optimize architectural choices for large language models, balancing accuracy and inference efficiency.
large language model LLM inference cost hidden size MLP attention mlp-to-attention ratio grouped-query attention GQA Chinchilla framework conditional scaling law inference-efficient accuracy training budget LLaMA-3.2