MaLA-500: Massive Language Adaptation of Large Language Models
Large language models have advanced the state of the art in natural language
processing. However, their predominant design for English or a limited set of
languages creates a substantial gap in their effectiveness for low-resource
languages. To bridge this gap, we introduce MaLA-500, a novel large language
model designed to cover an extensive range of 534 languages. To train MaLA-500,
we employ vocabulary extension and continued pretraining on LLaMA 2 with
Glot500-c. Our experiments on SIB-200 show that MaLA-500 achieves
state-of-the-art in-context learning results. We release MaLA-500 at
https://huggingface.co/MaLA-LM
65b1e894b4225e3e42341d1b
Peiqin Lin
admin_assigned
2024-01-25T09:18:16.574Z
6347e493e3f2de18545cbfe8
https://cdn-avatars.huggingface.co/v1/production/uploads/6347e493e3f2de18545cbfe8/_Y068r6Qh7QxqqntRDqC-.jpeg
Peiqin Lin
user
lpq29743
65b1e894b4225e3e42341d1c
Shaoxiong Ji
admin_assigned
2024-01-25T09:20:12.959Z
617a92e16f37340367d5d791
https://cdn-avatars.huggingface.co/v1/production/uploads/617a92e16f37340367d5d791/omgyzmaF90KBLa3YgFxhS.png
Shaoxiong
user
jisx
65b1e894b4225e3e42341d1d
Jörg Tiedemann
admin_assigned
2024-01-25T09:18:40.533Z
5eac04968a595438195ab4a6
/avatars/0223584657947ca7393ba46d211d6970.svg
Jörg Tiedemann
user
tiedeman
65b1e894b4225e3e42341d1e
André F. T. Martins
admin_assigned
2024-01-25T09:22:09.339Z
6326de72a28c096b45a8f8da
/avatars/e88e756fe2d193667b3bee839287ef51.svg
Andre Martins
user
andre-martins
65b1e894b4225e3e42341d1f
Hinrich Schütze
MaLA-500: Massive Language Adaptation of Large Language Models
Large language models have advanced the state of the art in natural language
processing. However, their predominant design for English or a limited set of
languages creates a substantial gap in their effectiveness for low-resource
languages. To bridge this gap, we introduce MaLA-500, a novel large language
model designed to cover an extensive range of 534 languages. To train MaLA-500,
we employ vocabulary extension and continued pretraining on LLaMA 2 with
Glot500-c. Our experiments on SIB-200 show that MaLA-500 achieves
state-of-the-art in-context learning results. We release MaLA-500 at
https://huggingface.co/MaLA-LM
MaLA-500, a large language model covering 534 languages, achieves state-of-the-art in-context learning by extending the LLaMA 2 vocabulary with Glot500-c.
MaLA-500 large language models in-context learning LLaMA 2 vocabulary extension Glot500-c