GEB-1.3B: Open Lightweight Large Language Model
Recently developed large language models (LLMs) such as ChatGPT, Claude, and
Llama have demonstrated impressive abilities, and even surpass human-level
performance in several tasks. Despite their success, the resource-intensive
demands of these models, requiring significant computational power for both
training and inference, limit their deployment to high-performance servers.
Additionally, the extensive calculation requirements of the models often lead
to increased latency in response times. With the increasing need for LLMs to
operate efficiently on CPUs, research about lightweight models that are
optimized for CPU inference has emerged. In this work, we introduce GEB-1.3B, a
lightweight LLM trained on 550 billion tokens in both Chinese and English
languages. We employ novel training techniques, including ROPE,
Group-Query-Attention, and FlashAttention-2, to accelerate training while
maintaining model performance. Additionally, we fine-tune the model using 10
million samples of instruction data to enhance alignment. GEB-1.3B exhibits
outstanding performance on general benchmarks such as MMLU, C-Eval, and CMMLU,
outperforming comparative models such as MindLLM-1.3B and TinyLLaMA-1.1B.
Notably, the FP32 version of GEB-1.3B achieves commendable inference times on
CPUs, with ongoing efforts to further enhance speed through advanced
quantization techniques. The release of GEB-1.3B as an open-source model marks
a significant contribution to the development of lightweight LLMs, promising to
foster further research and innovation in the field.
666ff529b07525f0bdec8aee
Jie Wu
claimed_verified
2024-06-21T07:07:13.898Z
6593be1ba78a2778033e1e1a
/avatars/f1b18cd1b4aa8f255b3b7f2a15cf08cf.svg
Wujie
user
Wu416732042
666ff529b07525f0bdec8aef
Yufeng Zhu
admin_assigned
2024-06-17T14:15:53.783Z
63464df0fe134dfd7a11f61d
/avatars/d880f1c66945e91db7336f50f5dd7e41.svg
zhu
user
yufengzhu
666ff529b07525f0bdec8af0
Lei Shen
claimed_verified
2024-06-18T13:28:44.698Z
6454d2ee1a543cf97b1ba671
/avatars/30edc30cf3684bf685aabffcec8a3c30.svg
shen
user
lorashen
666ff529b07525f0bdec8af1
Xuqing Lu
admin_assigned
2024-06-17T14:16:00.236Z
666a94e8cf27eed0f226cf2d
/avatars/282f548c26c62660c9cd6b8d806ebe92.svg
xuqing lu
user
luxq
GEB-1.3B: Open Lightweight Large Language Model
Recently developed large language models (LLMs) such as ChatGPT, Claude, and
Llama have demonstrated impressive abilities, and even surpass human-level
performance in several tasks. Despite their success, the resource-intensive
demands of these models, requiring significant computational power for both
training and inference, limit their deployment to high-performance servers.
Additionally, the extensive calculation requirements of the models often lead
to increased latency in response times. With the increasing need for LLMs to
operate efficiently on CPUs, research about lightweight models that are
optimized for CPU inference has emerged. In this work, we introduce GEB-1.3B, a
lightweight LLM trained on 550 billion tokens in both Chinese and English
languages. We employ novel training techniques, including ROPE,
Group-Query-Attention, and FlashAttention-2, to accelerate training while
maintaining model performance. Additionally, we fine-tune the model using 10
million samples of instruction data to enhance alignment. GEB-1.3B exhibits
outstanding performance on general benchmarks such as MMLU, C-Eval, and CMMLU,
outperforming comparative models such as MindLLM-1.3B and TinyLLaMA-1.1B.
Notably, the FP32 version of GEB-1.3B achieves commendable inference times on
CPUs, with ongoing efforts to further enhance speed through advanced
quantization techniques. The release of GEB-1.3B as an open-source model marks
a significant contribution to the development of lightweight LLMs, promising to
foster further research and innovation in the field.
GEB-1.3B, a lightweight LLM trained on multilingual data, employs novel techniques to achieve efficient CPU inference and outperforms other lightweight models on benchmarks.
large language models LLMs ChatGPT Claude Llama GEB-1.3B lightweight models CPU inference ROPE Group-Query-Attention FlashAttention-2 fine-tuning MMLU C-Eval CMMLU MindLLM-1.3B TinyLLaMA-1.1B FP32 quantization techniques