Speed Always Wins: A Survey on Efficient Architectures for Large
  Language Models
Large Language Models (LLMs) have delivered impressive results in language
understanding, generation, reasoning, and pushes the ability boundary of
multimodal models. Transformer models, as the foundation of modern LLMs, offer
a strong baseline with excellent scaling properties. However, the traditional
transformer architecture requires substantial computations and poses
significant obstacles for large-scale training and practical deployment. In
this survey, we offer a systematic examination of innovative LLM architectures
that address the inherent limitations of transformers and boost the efficiency.
Starting from language modeling, this survey covers the background and
technical details of linear and sparse sequence modeling methods, efficient
full attention variants, sparse mixture-of-experts, hybrid model architectures
incorporating the above techniques, and emerging diffusion LLMs. Additionally,
we discuss applications of these techniques to other modalities and consider
their wider implications for developing scalable, resource-aware foundation
models. By grouping recent studies into the above category, this survey
presents a blueprint of modern efficient LLM architectures, and we hope this
could help motivate future research toward more efficient, versatile AI
systems.
68a31244b65388761d074306
Weigao Sun
claimed_verified
2025-08-20T08:53:05.742Z
6246bb33da617c00b48e4d92
/avatars/0304a9f6eb7f5dee4d933d03222f94e9.svg
Weigao Sun
user
weigao266
68a31244b65388761d074307
Jiaxi Hu
68a31244b65388761d074308
Yucheng Zhou
68a31244b65388761d074309
Jusen Du
68a31244b65388761d07430a
Disen Lan
claimed_verified
2025-08-19T08:05:04.185Z
66ea643899af9ac3463639b1
/avatars/252d470e761a57834dee3dbc60dfefed.svg
Disen Lan
user
landisen
68a31244b65388761d07430b
Kexin Wang
68a31244b65388761d07430c
Tong Zhu
68a31244b65388761d07430d
Xiaoye Qu
68a31244b65388761d07430e
Yu Zhang
68a31244b65388761d07430f
Xiaoyu Mo
68a31244b65388761d074310
Daizong Liu
68a31244b65388761d074311
Yuxuan Liang
68a31244b65388761d074312
Wenliang Chen
68a31244b65388761d074313
Guoqi Li
68a31244b65388761d074314
Yu Cheng
Speed Always Wins: A Survey on Efficient Architectures for Large
  Language Models
Large Language Models (LLMs) have delivered impressive results in language
understanding, generation, reasoning, and pushes the ability boundary of
multimodal models. Transformer models, as the foundation of modern LLMs, offer
a strong baseline with excellent scaling properties. However, the traditional
transformer architecture requires substantial computations and poses
significant obstacles for large-scale training and practical deployment. In
this survey, we offer a systematic examination of innovative LLM architectures
that address the inherent limitations of transformers and boost the efficiency.
Starting from language modeling, this survey covers the background and
technical details of linear and sparse sequence modeling methods, efficient
full attention variants, sparse mixture-of-experts, hybrid model architectures
incorporating the above techniques, and emerging diffusion LLMs. Additionally,
we discuss applications of these techniques to other modalities and consider
their wider implications for developing scalable, resource-aware foundation
models. By grouping recent studies into the above category, this survey
presents a blueprint of modern efficient LLM architectures, and we hope this
could help motivate future research toward more efficient, versatile AI
systems.
This survey examines innovative architectures for large language models to enhance efficiency, covering linear and sparse sequence modeling, efficient attention mechanisms, sparse mixture-of-experts, hybrid models, and diffusion LLMs.
transformer models linear sequence modeling sparse sequence modeling efficient full attention sparse mixture-of-experts hybrid model architectures diffusion LLMs