PolyLM: An Open Source Polyglot Large Language Model
Large language models (LLMs) demonstrate remarkable ability to comprehend,
reason, and generate following nature language instructions. However, the
development of LLMs has been primarily focused on high-resource languages, such
as English, thereby limiting their applicability and research in other
languages. Consequently, we present PolyLM, a multilingual LLM trained on 640
billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its
multilingual capabilities, we 1) integrate bilingual data into training data;
and 2) adopt a curriculum learning strategy that increases the proportion of
non-English data from 30% in the first stage to 60% in the final stage during
pre-training. Further, we propose a multilingual self-instruct method which
automatically generates 132.7K diverse multilingual instructions for model
fine-tuning. To assess the model's performance, we collect several existing
multilingual tasks, including multilingual understanding, question answering,
generation, and translation. Extensive experiments show that PolyLM surpasses
other open-source models such as LLaMA and BLOOM on multilingual tasks while
maintaining comparable performance in English. Our models, alone with the
instruction data and multilingual benchmark, are available at:
https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation.
64af6223ddf1f172eb2d6770
Xiangpeng Wei
claimed_verified
2023-07-13T08:32:03.355Z
64abad4d329f5e6e7831ced4
/avatars/7702da77650df0be9f21cc300f761975.svg
Xiangpeng Wei
user
pemywei
64af6223ddf1f172eb2d6771
Haoran Wei
admin_assigned
2023-07-13T10:14:01.496Z
628b430afc0078a72e38b04a
/avatars/b4958c184be06534645f2284635d850e.svg
Haoran Wei
user
whr94621
64af6223ddf1f172eb2d6772
Huan Lin
64af6223ddf1f172eb2d6773
Tianhao Li
admin_assigned
2023-07-13T10:16:19.015Z
64abb96bd691b1c2482e7c19
/avatars/ff3a429f985a52c9c5ea4f64872599f2.svg
litianhao
user
litianhao
64af6223ddf1f172eb2d6774
Pei Zhang
admin_assigned
2023-07-13T10:16:49.434Z
641190aebeea38a6513bc558
/avatars/ff2f97adf98861de563017d3589961f1.svg
Pei Zhang
user
psyangqi
64af6223ddf1f172eb2d6775
Xingzhang Ren
claimed_verified
2025-01-03T14:07:15.564Z
645c82c4081be4b32049633a
/avatars/e5a08cf0ec5a04bd9d66111382ce0508.svg
xzhren
user
xingzhang
64af6223ddf1f172eb2d6776
Mei Li
64af6223ddf1f172eb2d6777
Yu Wan
64af6223ddf1f172eb2d6778
Zhiwei Cao
64af6223ddf1f172eb2d6779
Binbin Xie
64af6223ddf1f172eb2d677a
Tianxiang Hu
64af6223ddf1f172eb2d677b
Shangjie Li
64af6223ddf1f172eb2d677c
Binyuan Hui
claimed_verified
2023-07-13T03:33:47.001Z
61e4c4ca1ab24785ac11ba69
https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
Binyuan Hui
user
huybery
64af6223ddf1f172eb2d677d
Bowen Yu
claimed_verified
2023-07-17T06:48:47.381Z
63d9d68c1cae35c27bf7a6a7
/avatars/b5ad98cf269ae5f1fe90861fb4170fae.svg
Bowen Yu
user
Tigerph
64af6223ddf1f172eb2d677e
Dayiheng Liu
claimed_verified
2024-12-20T08:34:52.376Z
6434d4989bd5a84b5dd0b0f5
https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
Dayiheng Liu
user
Losin94
64af6223ddf1f172eb2d677f
Baosong Yang
64af6223ddf1f172eb2d6780
Fei Huang
64af6223ddf1f172eb2d6781
Jun Xie
PolyLM: An Open Source Polyglot Large Language Model
Large language models (LLMs) demonstrate remarkable ability to comprehend,
reason, and generate following nature language instructions. However, the
development of LLMs has been primarily focused on high-resource languages, such
as English, thereby limiting their applicability and research in other
languages. Consequently, we present PolyLM, a multilingual LLM trained on 640
billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its
multilingual capabilities, we 1) integrate bilingual data into training data;
and 2) adopt a curriculum learning strategy that increases the proportion of
non-English data from 30% in the first stage to 60% in the final stage during
pre-training. Further, we propose a multilingual self-instruct method which
automatically generates 132.7K diverse multilingual instructions for model
fine-tuning. To assess the model's performance, we collect several existing
multilingual tasks, including multilingual understanding, question answering,
generation, and translation. Extensive experiments show that PolyLM surpasses
other open-source models such as LLaMA and BLOOM on multilingual tasks while
maintaining comparable performance in English. Our models, alone with the
instruction data and multilingual benchmark, are available at:
https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation.
PolyLM, a multilingual LLM trained on 640 billion tokens, enhances multilingual capabilities through bilingual data and curriculum learning, outperforming other models on multilingual tasks while maintaining English performance.
large language models multilingual LLM bilingual data curriculum learning multilingual self-instruct multilingual understanding question answering generation translation