Tele-FLM Technical Report
Large language models (LLMs) have showcased profound capabilities in language
understanding and generation, facilitating a wide array of applications.
However, there is a notable paucity of detailed, open-sourced methodologies on
efficiently scaling LLMs beyond 50 billion parameters with minimum
trial-and-error cost and computational resources. In this report, we introduce
Tele-FLM (aka FLM-2), a 52B open-sourced multilingual large language model that
features a stable, efficient pre-training paradigm and enhanced factual
judgment capabilities. Tele-FLM demonstrates superior multilingual language
modeling abilities, measured by BPB on textual corpus. Besides, in both English
and Chinese foundation model evaluation, it is comparable to strong
open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B
and DeepSeek-67B. In addition to the model weights, we share the core designs,
engineering practices, and training details, which we expect to benefit both
the academic and industrial communities.
662b17cee5d910dd729834f9
Xiang Li
claimed_verified
2024-04-26T10:05:34.081Z
6486ea539264d020ddb2a06e
/avatars/3d5f2e8e13c87ad89197f171395d9556.svg
Xiang Li
user
sleepylx
662b17cee5d910dd729834fa
Yiqun Yao
admin_assigned
2024-04-26T09:45:35.744Z
64f9b61f0c7dbab99d9b0c16
/avatars/fe5e09578cdebc97c970e6795c836a49.svg
Yiqun Yao
user
yqyao
662b17cee5d910dd729834fb
Xin Jiang
admin_assigned
2024-04-26T09:45:55.068Z
647415007afa69c3c7a98f1f
https://cdn-avatars.huggingface.co/v1/production/uploads/647415007afa69c3c7a98f1f/pEl0PozmzNK8_PwUMiikd.jpeg
Xin Jiang
user
horiz94
662b17cee5d910dd729834fc
Xuezhi Fang
claimed_verified
2024-04-28T19:30:11.025Z
63c4f5588d95a5c770700ff8
/avatars/4f64029f74cba179d263c0e4f1887427.svg
Jason Fang
user
jasonfang3900
662b17cee5d910dd729834fd
Chao Wang
662b17cee5d910dd729834fe
Xinzhang Liu
662b17cee5d910dd729834ff
Zihan Wang
662b17cee5d910dd72983500
Yu Zhao
662b17cee5d910dd72983501
Xin Wang
662b17cee5d910dd72983502
Yuyao Huang
662b17cee5d910dd72983503
Shuangyong Song
662b17cee5d910dd72983504
Yongxiang Li
admin_assigned
2024-04-26T10:03:54.318Z
660b9ef3f4ab651901147194
/avatars/12720ddf21084f9ec48313d156918d63.svg
yongxiang
user
liyongxiang
662b17cee5d910dd72983505
Zheng Zhang
662b17cee5d910dd72983506
Bo Zhao
admin_assigned
2024-04-26T10:06:14.593Z
64acafac1aee69ece03af05a
/avatars/6380777852546dbd432fed5544d5db0b.svg
Bo Zhao
user
Z-LIRA
662b17cee5d910dd72983507
Aixin Sun
662b17cee5d910dd72983508
Yequan Wang
admin_assigned
2024-04-26T10:06:38.480Z
64f6db81867bbeac42b22b5d
/avatars/cb945f3fce8f3b77f0316c83b1496a29.svg
Wang Yequan
user
thuwyq
662b17cee5d910dd72983509
Zhongjiang He
662b17cee5d910dd7298350a
Zhongyuan Wang
662b17cee5d910dd7298350b
Xuelong Li
662b17cee5d910dd7298350c
Tiejun Huang
Tele-FLM Technical Report
Large language models (LLMs) have showcased profound capabilities in language
understanding and generation, facilitating a wide array of applications.
However, there is a notable paucity of detailed, open-sourced methodologies on
efficiently scaling LLMs beyond 50 billion parameters with minimum
trial-and-error cost and computational resources. In this report, we introduce
Tele-FLM (aka FLM-2), a 52B open-sourced multilingual large language model that
features a stable, efficient pre-training paradigm and enhanced factual
judgment capabilities. Tele-FLM demonstrates superior multilingual language
modeling abilities, measured by BPB on textual corpus. Besides, in both English
and Chinese foundation model evaluation, it is comparable to strong
open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B
and DeepSeek-67B. In addition to the model weights, we share the core designs,
engineering practices, and training details, which we expect to benefit both
the academic and industrial communities.
large language models (LLMs) pre-training paradigm BPB (bits per byte) textual corpus English foundation model Chinese foundation model Llama2-70B DeepSeek-67B