Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model
In this study, we introduce CT-LLM, a 2B large language model (LLM) that
illustrates a pivotal shift towards prioritizing the Chinese language in
developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the
conventional methodology by primarily incorporating Chinese textual data,
utilizing an extensive corpus of 1,200 billion tokens, including 800 billion
Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This
strategic composition facilitates the model's exceptional proficiency in
understanding and processing Chinese, a capability further enhanced through
alignment techniques. Demonstrating remarkable performance on the CHC-Bench,
CT-LLM excels in Chinese language tasks, and showcases its adeptness in English
through SFT. This research challenges the prevailing paradigm of training LLMs
predominantly on English corpora and then adapting them to other languages,
broadening the horizons for LLM training methodologies. By open-sourcing the
full process of training a Chinese LLM, including a detailed data processing
procedure with the obtained Massive Appropriate Pretraining Chinese Corpus
(MAP-CC), a well-chosen multidisciplinary Chinese Hard Case Benchmark
(CHC-Bench), and the 2B-size Chinese Tiny LLM (CT-LLM), we aim to foster
further exploration and innovation in both academia and industry, paving the
way for more inclusive and versatile language models.
66135b40db728ea967146837
Xinrun Du
66135b40db728ea967146838
Zhouliang Yu
claimed_verified
2025-03-02T20:20:03.663Z
62a80fe3ac97233f1625235a
https://cdn-avatars.huggingface.co/v1/production/uploads/62a80fe3ac97233f1625235a/_rGtpqdY7OEBz3pyqb6fE.jpeg
Zhouliang Yu
user
zhouliang
66135b40db728ea967146839
Songyang Gao
66135b40db728ea96714683a
Ding Pan
66135b40db728ea96714683b
Yuyang Cheng
66135b40db728ea96714683c
Ziyang Ma
66135b40db728ea96714683d
Ruibin Yuan
claimed_verified
2025-03-12T08:57:35.037Z
5fd6f670053c8345eddc1b68
https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
Ruibin Yuan
user
a43992899
66135b40db728ea96714683e
Xingwei Qu
66135b40db728ea96714683f
Jiaheng Liu
claimed_verified
2025-01-05T23:03:56.063Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
66135b40db728ea967146840
Tianyu Zheng
claimed_verified
2024-10-21T07:54:26.941Z
64ab99dcb76bfd863eba64c1
https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
TY.Zheng
user
aaabiao
66135b40db728ea967146841
Xinchen Luo
66135b40db728ea967146842
Guorui Zhou
66135b40db728ea967146843
Binhang Yuan
66135b40db728ea967146844
Wenhu Chen
66135b40db728ea967146845
Jie Fu
66135b40db728ea967146846
Ge Zhang
claimed_verified
2024-05-31T06:53:38.464Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model
In this study, we introduce CT-LLM, a 2B large language model (LLM) that
illustrates a pivotal shift towards prioritizing the Chinese language in
developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the
conventional methodology by primarily incorporating Chinese textual data,
utilizing an extensive corpus of 1,200 billion tokens, including 800 billion
Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This
strategic composition facilitates the model's exceptional proficiency in
understanding and processing Chinese, a capability further enhanced through
alignment techniques. Demonstrating remarkable performance on the CHC-Bench,
CT-LLM excels in Chinese language tasks, and showcases its adeptness in English
through SFT. This research challenges the prevailing paradigm of training LLMs
predominantly on English corpora and then adapting them to other languages,
broadening the horizons for LLM training methodologies. By open-sourcing the
full process of training a Chinese LLM, including a detailed data processing
procedure with the obtained Massive Appropriate Pretraining Chinese Corpus
(MAP-CC), a well-chosen multidisciplinary Chinese Hard Case Benchmark
(CHC-Bench), and the 2B-size Chinese Tiny LLM (CT-LLM), we aim to foster
further exploration and innovation in both academia and industry, paving the
way for more inclusive and versatile language models.
CT-LLM, a large language model emphasizing Chinese language training, demonstrates exceptional proficiency through extensive Chinese textual data and outperforms in both Chinese and English tasks.
large language model (LLM) Chinese language CHC-Bench SFT MAP-CC Chinese Tiny LLM (CT-LLM)