A Survey of Small Language Models
Small Language Models (SLMs) have become increasingly important due to their
efficiency and performance to perform various language tasks with minimal
computational resources, making them ideal for various settings including
on-device, mobile, edge devices, among many others. In this article, we present
a comprehensive survey on SLMs, focusing on their architectures, training
techniques, and model compression techniques. We propose a novel taxonomy for
categorizing the methods used to optimize SLMs, including model compression,
pruning, and quantization techniques. We summarize the benchmark datasets that
are useful for benchmarking SLMs along with the evaluation metrics commonly
used. Additionally, we highlight key open challenges that remain to be
addressed. Our survey aims to serve as a valuable resource for researchers and
practitioners interested in developing and deploying small yet efficient
language models.
672043886e75d23f3e2141db
Chien Van Nguyen
claimed_verified
2024-10-30T09:54:24.455Z
63edf17df5aef7e657262605
https://cdn-avatars.huggingface.co/v1/production/uploads/63edf17df5aef7e657262605/X3WzygIyND_0T_SWRDDU4.jpeg
Nguyen Van Chien
user
chiennv
672043886e75d23f3e2141dc
Xuan Shen
claimed_verified
2025-03-25T08:22:32.667Z
65f7197c66afd39dd7f58522
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zx-6RlvxTATmIO3dCIYez.png
Shawn Shen
user
shawnricecake
672043886e75d23f3e2141dd
Ryan Aponte
672043886e75d23f3e2141de
Yu Xia
672043886e75d23f3e2141df
Samyadeep Basu
claimed_verified
2024-10-31T18:14:25.665Z
6384d891cafe4e4ac4060206
/avatars/45eb04bf6e0ed1a4daa14070411b693f.svg
Samyadeep
user
samyadeepbasu
672043886e75d23f3e2141e0
Zhengmian Hu
672043886e75d23f3e2141e1
Jian Chen
claimed_verified
2024-11-15T09:39:18.576Z
64593836c16ecb4815e082ce
https://cdn-avatars.huggingface.co/v1/production/uploads/64593836c16ecb4815e082ce/T9WM0Ly6ADi3a9JUSIsSV.png
Jian Chen
user
puar-playground
672043886e75d23f3e2141e2
Mihir Parmar
672043886e75d23f3e2141e3
Sasidhar Kunapuli
672043886e75d23f3e2141e4
Joe Barrow
672043886e75d23f3e2141e5
Junda Wu
672043886e75d23f3e2141e6
Ashish Singh
672043886e75d23f3e2141e7
Yu Wang
672043886e75d23f3e2141e8
Jiuxiang Gu
672043886e75d23f3e2141e9
Franck Dernoncourt
claimed_verified
2024-10-31T09:55:11.391Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
672043886e75d23f3e2141ea
Nesreen K. Ahmed
claimed_verified
2025-01-07T08:43:09.275Z
663e588ccca86c8371a913d9
https://cdn-avatars.huggingface.co/v1/production/uploads/663e588ccca86c8371a913d9/wgVGw5U77cqwtjovhsXyR.jpeg
Nesreen Ahmed
user
nkahmed
672043886e75d23f3e2141eb
Nedim Lipka
672043886e75d23f3e2141ec
Ruiyi Zhang
672043886e75d23f3e2141ed
Xiang Chen
672043886e75d23f3e2141ee
Tong Yu
672043886e75d23f3e2141ef
Sungchul Kim
672043886e75d23f3e2141f0
Hanieh Deilamsalehy
672043886e75d23f3e2141f1
Namyong Park
672043886e75d23f3e2141f2
Mike Rimer
672043886e75d23f3e2141f3
Zhehao Zhang
672043886e75d23f3e2141f4
Huanrui Yang
672043886e75d23f3e2141f5
Ryan A. Rossi
672043886e75d23f3e2141f6
Thien Huu Nguyen
A Survey of Small Language Models
Small Language Models (SLMs) have become increasingly important due to their
efficiency and performance to perform various language tasks with minimal
computational resources, making them ideal for various settings including
on-device, mobile, edge devices, among many others. In this article, we present
a comprehensive survey on SLMs, focusing on their architectures, training
techniques, and model compression techniques. We propose a novel taxonomy for
categorizing the methods used to optimize SLMs, including model compression,
pruning, and quantization techniques. We summarize the benchmark datasets that
are useful for benchmarking SLMs along with the evaluation metrics commonly
used. Additionally, we highlight key open challenges that remain to be
addressed. Our survey aims to serve as a valuable resource for researchers and
practitioners interested in developing and deploying small yet efficient
language models.
A survey of small language models covers their architectures, training methods, and model compression techniques, with a taxonomy for optimization, benchmark datasets, evaluation metrics, and open challenges.
architectures training techniques model compression pruning quantization techniques benchmark datasets evaluation metrics small language models