MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT
"Bigger the better" has been the predominant trend in recent Large Language
Models (LLMs) development. However, LLMs do not suit well for scenarios that
require on-device processing, energy efficiency, low memory footprint, and
response efficiency. These requisites are crucial for privacy, security, and
sustainable deployment. This paper explores the "less is more" paradigm by
addressing the challenge of designing accurate yet efficient Small Language
Models (SLMs) for resource constrained devices. Our primary contribution is the
introduction of an accurate and fully transparent open-source 0.5 billion
(0.5B) parameter SLM, named MobiLlama, catering to the specific needs of
resource-constrained computing with an emphasis on enhanced performance with
reduced resource demands. MobiLlama is a SLM design that initiates from a
larger model and applies a careful parameter sharing scheme to reduce both the
pre-training and the deployment cost. Our work strives to not only bridge the
gap in open-source SLMs but also ensures full transparency, where complete
training data pipeline, training code, model weights, and over 300 checkpoints
along with evaluation codes is available at :
https://github.com/mbzuai-oryx/MobiLlama.
65dd76e84a4fce1ec96347f4
Omkar Thawakar
admin_assigned
2024-02-27T10:29:58.983Z
64b7d2ad8c632fbca9507431
/avatars/76c31ea218108cf6c3715269f7605404.svg
Omkar Thawakar
user
omkarthawakar
65dd76e84a4fce1ec96347f5
Ashmal Vayani
admin_assigned
2024-02-27T10:30:14.795Z
64f9d0ded88459d40bd0efb9
https://cdn-avatars.huggingface.co/v1/production/uploads/64f9d0ded88459d40bd0efb9/16Y41IUX0T_TZj_g_tTBF.jpeg
Ashmal Vayani
user
Ashmal
65dd76e84a4fce1ec96347f6
Salman Khan
65dd76e84a4fce1ec96347f7
Hisham Cholakal
admin_assigned
2024-02-27T10:30:47.610Z
654a5f4f9b8bd6406d45bb46
/avatars/ac0d7eef62cd98a280b162cf7896b1a2.svg
Hisham Cholakkal
user
hishamcholakkal
65dd76e84a4fce1ec96347f8
Rao M. Anwer
65dd76e84a4fce1ec96347f9
Michael Felsberg
admin_assigned
2024-02-27T10:31:08.550Z
63d38a1a9e1b81818c816905
/avatars/a2160bb9fbb8a8eec129b8eab559294b.svg
Michael Felsberg
user
CVL-LiU
65dd76e84a4fce1ec96347fa
Tim Baldwin
admin_assigned
2024-02-27T10:31:14.438Z
65d895d8d435dce78392065f
/avatars/68c7185d3dd1adf78927b91849978d62.svg
Tim Baldwin
user
tbaldwin
65dd76e84a4fce1ec96347fb
Eric P. Xing
admin_assigned
2024-02-27T10:31:25.777Z
64ff67722ad36636be6c4542
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
Eric Xing
user
EricX003
65dd76e84a4fce1ec96347fc
Fahad Shahbaz Khan
MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT
"Bigger the better" has been the predominant trend in recent Large Language
Models (LLMs) development. However, LLMs do not suit well for scenarios that
require on-device processing, energy efficiency, low memory footprint, and
response efficiency. These requisites are crucial for privacy, security, and
sustainable deployment. This paper explores the "less is more" paradigm by
addressing the challenge of designing accurate yet efficient Small Language
Models (SLMs) for resource constrained devices. Our primary contribution is the
introduction of an accurate and fully transparent open-source 0.5 billion
(0.5B) parameter SLM, named MobiLlama, catering to the specific needs of
resource-constrained computing with an emphasis on enhanced performance with
reduced resource demands. MobiLlama is a SLM design that initiates from a
larger model and applies a careful parameter sharing scheme to reduce both the
pre-training and the deployment cost. Our work strives to not only bridge the
gap in open-source SLMs but also ensures full transparency, where complete
training data pipeline, training code, model weights, and over 300 checkpoints
along with evaluation codes is available at :
https://github.com/mbzuai-oryx/MobiLlama.
MobiLlama is a 0.5 billion parameter small language model designed for resource-constrained devices using parameter sharing to balance accuracy and efficiency.
Large Language Models Small Language Models resource constrained devices parameter sharing MobiLlama pre-training deployment cost full transparency training data pipeline training code model weights checkpoints evaluation codes