Skywork: A More Open Bilingual Foundation Model
In this technical report, we present Skywork-13B, a family of large language
models (LLMs) trained on a corpus of over 3.2 trillion tokens drawn from both
English and Chinese texts. This bilingual foundation model is the most
extensively trained and openly published LLMs of comparable size to date. We
introduce a two-stage training methodology using a segmented corpus, targeting
general purpose training and then domain-specific enhancement training,
respectively. We show that our model not only excels on popular benchmarks, but
also achieves state of the art performance in Chinese language modeling
on diverse domains. Furthermore, we propose a novel leakage detection method,
demonstrating that test data contamination is a pressing issue warranting
further investigation by the LLM community. To spur future research, we release
Skywork-13B along with checkpoints obtained during intermediate stages of the
training process. We are also releasing part of our SkyPile corpus, a
collection of over 150 billion tokens of web text, which is the largest high
quality open Chinese pre-training corpus to date. We hope Skywork-13B and our
open corpus will serve as a valuable open-source resource to democratize access
to high-quality LLMs.
65408e8aff86086e115360b3
Tianwen Wei
claimed_verified
2023-10-31T15:04:54.859Z
63d8773a635499f21644601b
/avatars/07efea204d172e2844dbacc418131da2.svg
Wei Tianwen
user
weitianwen
65408e8aff86086e115360b4
Liang Zhao
claimed_verified
2024-07-12T07:42:15.403Z
64996ce2ed09df36eaf236ea
https://cdn-avatars.huggingface.co/v1/production/uploads/64996ce2ed09df36eaf236ea/wIr1j2zkAezXfpsczTjq_.jpeg
zhaoliang
user
zhao1iang
65408e8aff86086e115360b5
Lichang Zhang
admin_assigned
2023-10-31T10:00:10.939Z
63f3396b4745321de34ed545
/avatars/f53c05b6ff1d59caeaf092853c04baf8.svg
lichang zhang
user
liczhang
65408e8aff86086e115360b6
Bo Zhu
65408e8aff86086e115360b7
Lijie Wang
65408e8aff86086e115360b8
Haihua Yang
admin_assigned
2023-10-31T10:00:54.659Z
653f24f4cd52f288490775e6
/avatars/4938e31c7dcbee80861ce948702e831a.svg
yanghaihua
user
yanghaihua
65408e8aff86086e115360b9
Biye Li
admin_assigned
2023-10-31T10:01:09.880Z
636a1719f8d9af4aea17095a
/avatars/5eb27d368aa18f242e91e2d6ed1c60cf.svg
Biye Li
user
dugusword
65408e8aff86086e115360ba
Cheng Cheng
claimed_verified
2024-06-12T09:31:22.366Z
64535b71bcbd25618f7655da
https://cdn-avatars.huggingface.co/v1/production/uploads/64535b71bcbd25618f7655da/vDc8nrYQd4rpYgXLWOuIx.jpeg
cheng cheng
user
chengtbf
65408e8aff86086e115360bb
Weiwei Lü
65408e8aff86086e115360bc
Rui Hu
65408e8aff86086e115360bd
Chenxia Li
admin_assigned
2023-10-31T10:02:24.788Z
62cec5bea3a23014aca6c3ee
/avatars/b15f262fad2d146481df4328ee42f78b.svg
lichenxia
user
lichenxia
65408e8aff86086e115360be
Liu Yang
65408e8aff86086e115360bf
Xilin Luo
65408e8aff86086e115360c0
Xuejie Wu
65408e8aff86086e115360c1
Lunan Liu
admin_assigned
2023-10-31T10:13:20.667Z
624eabc864599731f224e87b
/avatars/c640d4168c359f6f75ffa50b60fb4d7b.svg
Lunan LIU
user
rangnide
65408e8aff86086e115360c2
Wenjun Cheng
65408e8aff86086e115360c3
Peng Cheng
claimed_verified
2024-06-12T09:24:32.179Z
636b8f4429668f04eaf5d015
/avatars/84b6209914cb83af7cfce2ffb337d7bd.svg
CPFLAME
user
CPFLAME
65408e8aff86086e115360c4
Jianhao Zhang
admin_assigned
2023-10-31T10:13:53.447Z
64235adf23b98d5ee6e9ca85
https://cdn-avatars.huggingface.co/v1/production/uploads/64235adf23b98d5ee6e9ca85/m_7uwLaAkTXyt3qB2pPLA.png
Jianhao Zhang
user
Heerozh
65408e8aff86086e115360c5
Xiaoyu Zhang
claimed_verified
2024-06-12T11:51:28.972Z
63647823a1503c61efd4fbcb
https://cdn-avatars.huggingface.co/v1/production/uploads/63647823a1503c61efd4fbcb/Df5U59Imobmr9S8wDf77I.png
Yi Wang
user
BBuf
65408e8aff86086e115360c6
Lei Lin
65408e8aff86086e115360c7
Xiaokun Wang
claimed_verified
2025-01-09T10:13:03.978Z
62be9b5aae56e75e4d689e7c
/avatars/6772bc09d6eeb4e86b1210481be91720.svg
wangxiaokun
user
shawn0wang
65408e8aff86086e115360c8
Yutuan Ma
65408e8aff86086e115360c9
Chuanhai Dong
65408e8aff86086e115360ca
Yanqi Sun
65408e8aff86086e115360cb
Yifu Chen
65408e8aff86086e115360cc
Yongyi Peng
65408e8aff86086e115360cd
Xiaojuan Liang
65408e8aff86086e115360ce
Shuicheng Yan
65408e8aff86086e115360cf
Han Fang
65408e8aff86086e115360d0
Yahui Zhou
Skywork: A More Open Bilingual Foundation Model
In this technical report, we present Skywork-13B, a family of large language
models (LLMs) trained on a corpus of over 3.2 trillion tokens drawn from both
English and Chinese texts. This bilingual foundation model is the most
extensively trained and openly published LLMs of comparable size to date. We
introduce a two-stage training methodology using a segmented corpus, targeting
general purpose training and then domain-specific enhancement training,
respectively. We show that our model not only excels on popular benchmarks, but
also achieves state of the art performance in Chinese language modeling
on diverse domains. Furthermore, we propose a novel leakage detection method,
demonstrating that test data contamination is a pressing issue warranting
further investigation by the LLM community. To spur future research, we release
Skywork-13B along with checkpoints obtained during intermediate stages of the
training process. We are also releasing part of our SkyPile corpus, a
collection of over 150 billion tokens of web text, which is the largest high
quality open Chinese pre-training corpus to date. We hope Skywork-13B and our
open corpus will serve as a valuable open-source resource to democratize access
to high-quality LLMs.
Skywork-13B, a bilingual LLM trained on extensive corpora, excels in benchmarks and achieves state-of-the-art performance in Chinese language modeling, introducing a novel leakage detection method.
large language models LLMs segmented corpus general purpose training domain-specific enhancement training state of the art Chinese language modeling leakage detection method SkyPile corpus