LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation
  Capabilities Beyond 100 Languages
Large Language Models~(LLMs) demonstrate remarkable translation capabilities
in high-resource language tasks, yet their performance in low-resource
languages is hindered by insufficient multilingual data during pre-training. To
address this, we dedicate 35,000 A100-SXM4-80GB GPU hours in conducting
extensive multilingual continual pre-training on the LLaMA series models,
enabling translation support across more than 100 languages. Through a
comprehensive analysis of training strategies, such as vocabulary expansion and
data augmentation, we develop LLaMAX. Remarkably, without sacrificing its
generalization ability, LLaMAX achieves significantly higher translation
performance compared to existing open-source LLMs~(by more than 10 spBLEU
points) and performs on-par with specialized translation model~(M2M-100-12B) on
the Flores-101 benchmark. Extensive experiments indicate that LLaMAX can serve
as a robust multilingual foundation model. The
code~\url{https://github.com/CONE-MT/LLaMAX/.} and
models~\url{https://huggingface.co/LLaMAX/.} are publicly available.
668cab02a249df5a139d0ec1
Yinquan Lu
668cab02a249df5a139d0ec2
Wenhao Zhu
claimed_verified
2024-07-09T09:18:40.502Z
649d7d8968586ca9bf7f5fe6
/avatars/b444240770d4025dea41871cf38126dc.svg
Wenhao Zhu
user
Wenhao97
668cab02a249df5a139d0ec3
Lei Li
668cab02a249df5a139d0ec4
Yu Qiao
668cab02a249df5a139d0ec5
Fei Yuan
admin_assigned
2024-07-09T08:27:51.394Z
65fed45b08d35929362dd651
https://cdn-avatars.huggingface.co/v1/production/uploads/65fed45b08d35929362dd651/KLMxsyRN6_HhCZP1iDw6K.png
FeiYuan
user
FeYuan
LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation
  Capabilities Beyond 100 Languages
Large Language Models~(LLMs) demonstrate remarkable translation capabilities
in high-resource language tasks, yet their performance in low-resource
languages is hindered by insufficient multilingual data during pre-training. To
address this, we dedicate 35,000 A100-SXM4-80GB GPU hours in conducting
extensive multilingual continual pre-training on the LLaMA series models,
enabling translation support across more than 100 languages. Through a
comprehensive analysis of training strategies, such as vocabulary expansion and
data augmentation, we develop LLaMAX. Remarkably, without sacrificing its
generalization ability, LLaMAX achieves significantly higher translation
performance compared to existing open-source LLMs~(by more than 10 spBLEU
points) and performs on-par with specialized translation model~(M2M-100-12B) on
the Flores-101 benchmark. Extensive experiments indicate that LLaMAX can serve
as a robust multilingual foundation model. The
code~\url{https://github.com/CONE-MT/LLaMAX/.} and
models~\url{https://huggingface.co/LLaMAX/.} are publicly available.
LLaMAX enhances multilingual translation by extensive continual pre-training, achieving superior performance in 100+ languages without sacrificing generalization.
Large Language Models LLama series multilingual continual pre-training vocabulary expansion data augmentation spBLEU Flores-101 benchmark M2M-100-12B multilingual foundation model