MobileLLM: Optimizing Sub-billion Parameter Language Models for
  On-Device Use Cases
This paper addresses the growing need for efficient large language models
(LLMs) on mobile devices, driven by increasing cloud costs and latency
concerns. We focus on designing top-quality LLMs with fewer than a billion
parameters, a practical choice for mobile deployment. Contrary to prevailing
belief emphasizing the pivotal role of data and parameter quantity in
determining model quality, our investigation underscores the significance of
model architecture for sub-billion scale LLMs. Leveraging deep and thin
architectures, coupled with embedding sharing and grouped-query attention
mechanisms, we establish a strong baseline network denoted as MobileLLM, which
attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M
state-of-the-art models. Additionally, we propose an immediate block-wise
weight sharing approach with no increase in model size and only marginal
latency overhead. The resultant models, denoted as MobileLLM-LS, demonstrate a
further accuracy enhancement of 0.7%/0.8% than MobileLLM 125M/350M. Moreover,
MobileLLM model family shows significant improvements compared to previous
sub-billion models on chat benchmarks, and demonstrates close correctness to
LLaMA-v2 7B in API calling tasks, highlighting the capability of small models
for common on-device use cases.
65dc007f2bc96e6f004a0a4f
Zechun Liu
admin_assigned
2024-02-26T10:16:15.652Z
620ae55fe190de4b8f27ba9d
/avatars/aaa2d0eadbcf96c2eb9059e3d73c2760.svg
Liu
user
Zechun
65dc007f2bc96e6f004a0a50
Changsheng Zhao
admin_assigned
2024-02-26T10:16:22.535Z
64d49ef30f76abaf363b88d6
/avatars/93b5cc51305ac88198cea1dad8104db2.svg
Changsheng Zhao
user
mikezhaocs
65dc007f2bc96e6f004a0a51
Forrest Iandola
admin_assigned
2024-02-26T10:16:29.946Z
5f1753f4925b9863e28ad4de
/avatars/e56d25a40f2c0e4e6696b9738877fce8.svg
Forrest Iandola
user
forresti
65dc007f2bc96e6f004a0a52
Chen Lai
65dc007f2bc96e6f004a0a53
Yuandong Tian
admin_assigned
2024-02-26T10:16:48.983Z
6344cf73ee1504dbcd5bdfe7
/avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
Yuandong Tian
user
tydsh
65dc007f2bc96e6f004a0a54
Igor Fedorov
admin_assigned
2024-02-26T10:16:54.965Z
646159d6051604bda02d6f05
/avatars/d606fee0ad542cc085697b12e673b180.svg
igor fedorov
user
igo77
65dc007f2bc96e6f004a0a55
Yunyang Xiong
admin_assigned
2024-02-26T10:17:03.222Z
65304b62e7535baecd85d080
/avatars/6e546c7d1414bd92c5a7c8d8c404de92.svg
Yunyang Xiong
user
yunyangx
65dc007f2bc96e6f004a0a56
Ernie Chang
65dc007f2bc96e6f004a0a57
Yangyang Shi
65dc007f2bc96e6f004a0a58
Raghuraman Krishnamoorthi
65dc007f2bc96e6f004a0a59
Liangzhen Lai
admin_assigned
2024-02-26T10:19:03.248Z
64b705c73240387159397c71
/avatars/da339f07eb58f58756b91eba9af557a1.svg
Liangzhen Lai
user
liangzhen-lai
65dc007f2bc96e6f004a0a5a
Vikas Chandra
admin_assigned
2024-02-26T10:19:24.981Z
6566681f7b5ed0735812af32
/avatars/424905abb2973954b0850c592743b6fb.svg
Vikas Chandra
user
vchandra
MobileLLM: Optimizing Sub-billion Parameter Language Models for
  On-Device Use Cases
This paper addresses the growing need for efficient large language models
(LLMs) on mobile devices, driven by increasing cloud costs and latency
concerns. We focus on designing top-quality LLMs with fewer than a billion
parameters, a practical choice for mobile deployment. Contrary to prevailing
belief emphasizing the pivotal role of data and parameter quantity in
determining model quality, our investigation underscores the significance of
model architecture for sub-billion scale LLMs. Leveraging deep and thin
architectures, coupled with embedding sharing and grouped-query attention
mechanisms, we establish a strong baseline network denoted as MobileLLM, which
attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M
state-of-the-art models. Additionally, we propose an immediate block-wise
weight sharing approach with no increase in model size and only marginal
latency overhead. The resultant models, denoted as MobileLLM-LS, demonstrate a
further accuracy enhancement of 0.7%/0.8% than MobileLLM 125M/350M. Moreover,
MobileLLM model family shows significant improvements compared to previous
sub-billion models on chat benchmarks, and demonstrates close correctness to
LLaMA-v2 7B in API calling tasks, highlighting the capability of small models
for common on-device use cases.
MobileLLM leverages deep, thin architectures and advanced techniques to achieve high accuracy in sub-billion parameter models suitable for mobile devices.
LLM MobileLLM MobileLLM-LS embedding sharing grouped-query attention sub-billion scale LLMs weight sharing chat benchmarks API calling tasks