FuxiTranyu: A Multilingual Large Language Model Trained with Balanced
  Data
Large language models (LLMs) have demonstrated prowess in a wide range of
tasks. However, many LLMs exhibit significant performance discrepancies between
high- and low-resource languages. To mitigate this challenge, we present
FuxiTranyu, an open-source multilingual LLM, which is designed to satisfy the
need of the research community for balanced and high-performing multilingual
capabilities. FuxiTranyu-8B, the base model with 8 billion parameters, is
trained from scratch on a meticulously balanced multilingual data repository
that contains 600 billion tokens covering 43 natural languages and 16
programming languages. In addition to the base model, we also develop two
instruction-tuned models: FuxiTranyu-8B-SFT that is fine-tuned on a diverse
multilingual instruction dataset, and FuxiTranyu-8B-DPO that is further refined
with DPO on a preference dataset for enhanced alignment ability. Extensive
experiments on a wide range of multilingual benchmarks demonstrate the
competitive performance of FuxiTranyu against existing multilingual LLMs, e.g.,
BLOOM-7B, PolyLM-13B, Llama-2-Chat-7B and Mistral-7B-Instruct. Interpretability
analyses at both the neuron and representation level suggest that FuxiTranyu is
able to learn consistent multilingual representations across different
languages. To promote further research into multilingual LLMs and their working
mechanisms, we release both the base and instruction-tuned FuxiTranyu models
together with 58 pretraining checkpoints at HuggingFace and Github.
66bc35369d958666bc9bf156
Haoran Sun
claimed_verified
2024-08-14T13:03:39.825Z
669c82a44bbe8ad52ea113a5
/avatars/809256e0586e05fa754047322d3ccc68.svg
Haoran Sun
user
hrsun15
66bc35369d958666bc9bf157
Renren Jin
66bc35369d958666bc9bf158
Shaoyang Xu
66bc35369d958666bc9bf159
Leiyu Pan
66bc35369d958666bc9bf15a
Supryadi
66bc35369d958666bc9bf15b
Menglong Cui
66bc35369d958666bc9bf15c
Jiangcun Du
66bc35369d958666bc9bf15d
Yikun Lei
66bc35369d958666bc9bf15e
Lei Yang
66bc35369d958666bc9bf15f
Ling Shi
66bc35369d958666bc9bf160
Juesi Xiao
66bc35369d958666bc9bf161
Shaolin Zhu
66bc35369d958666bc9bf162
Deyi Xiong
FuxiTranyu: A Multilingual Large Language Model Trained with Balanced
  Data
Large language models (LLMs) have demonstrated prowess in a wide range of
tasks. However, many LLMs exhibit significant performance discrepancies between
high- and low-resource languages. To mitigate this challenge, we present
FuxiTranyu, an open-source multilingual LLM, which is designed to satisfy the
need of the research community for balanced and high-performing multilingual
capabilities. FuxiTranyu-8B, the base model with 8 billion parameters, is
trained from scratch on a meticulously balanced multilingual data repository
that contains 600 billion tokens covering 43 natural languages and 16
programming languages. In addition to the base model, we also develop two
instruction-tuned models: FuxiTranyu-8B-SFT that is fine-tuned on a diverse
multilingual instruction dataset, and FuxiTranyu-8B-DPO that is further refined
with DPO on a preference dataset for enhanced alignment ability. Extensive
experiments on a wide range of multilingual benchmarks demonstrate the
competitive performance of FuxiTranyu against existing multilingual LLMs, e.g.,
BLOOM-7B, PolyLM-13B, Llama-2-Chat-7B and Mistral-7B-Instruct. Interpretability
analyses at both the neuron and representation level suggest that FuxiTranyu is
able to learn consistent multilingual representations across different
languages. To promote further research into multilingual LLMs and their working
mechanisms, we release both the base and instruction-tuned FuxiTranyu models
together with 58 pretraining checkpoints at HuggingFace and Github.
FuxiTranyu, a multilingual large language model, achieves competitive performance across various benchmarks and learns consistent multilingual representations through balanced training and diverse fine-tuning.
multilingual LLM FuxiTranyu FuxiTranyu-8B instruction-tuned models FuxiTranyu-8B-SFT FuxiTranyu-8B-DPO multilingual data repository DPO preference dataset multilingual benchmarks consistent multilingual representations