Multilingual Instruction Tuning With Just a Pinch of Multilinguality
As instruction-tuned large language models (LLMs) gain global adoption, their
ability to follow instructions in multiple languages becomes increasingly
crucial. One promising approach is cross-lingual transfer, where a model
acquires specific functionality on some language by finetuning on another
language. In this work, we investigate how multilinguality during instruction
tuning of a multilingual LLM affects instruction-following across languages. We
first show that many languages transfer some instruction-following capabilities
to other languages from even monolingual tuning. Furthermore, we find that only
40 multilingual examples in an English tuning set substantially improve
multilingual instruction-following, both in seen and unseen languages during
tuning. In general, we observe that models tuned on multilingual mixtures
exhibit comparable or superior performance in several languages compared to
monolingually tuned models, despite training on 10x fewer examples in those
languages. Finally, we find that increasing the number of languages in the
instruction tuning set from 1 to only 2, 3, or 4 increases cross-lingual
generalization. Our results suggest that building massively multilingual
instruction-tuned models can be done with only a very small set of multilingual
instruction-responses.
659614d450a303befb4668bf
Uri Shaham
claimed_verified
2024-01-11T11:36:10.741Z
61265535ed8ae1abb08326d4
https://cdn-avatars.huggingface.co/v1/production/uploads/1629996654743-61265535ed8ae1abb08326d4.jpeg
Uri Shaham
user
Uri
659614d450a303befb4668c0
Jonathan Herzig
659614d450a303befb4668c1
Roee Aharoni
659614d450a303befb4668c2
Idan Szpektor
659614d450a303befb4668c3
Reut Tsarfaty
659614d450a303befb4668c4
Matan Eyal
Multilingual Instruction Tuning With Just a Pinch of Multilinguality
As instruction-tuned large language models (LLMs) gain global adoption, their
ability to follow instructions in multiple languages becomes increasingly
crucial. One promising approach is cross-lingual transfer, where a model
acquires specific functionality on some language by finetuning on another
language. In this work, we investigate how multilinguality during instruction
tuning of a multilingual LLM affects instruction-following across languages. We
first show that many languages transfer some instruction-following capabilities
to other languages from even monolingual tuning. Furthermore, we find that only
40 multilingual examples in an English tuning set substantially improve
multilingual instruction-following, both in seen and unseen languages during
tuning. In general, we observe that models tuned on multilingual mixtures
exhibit comparable or superior performance in several languages compared to
monolingually tuned models, despite training on 10x fewer examples in those
languages. Finally, we find that increasing the number of languages in the
instruction tuning set from 1 to only 2, 3, or 4 increases cross-lingual
generalization. Our results suggest that building massively multilingual
instruction-tuned models can be done with only a very small set of multilingual
instruction-responses.
Multilingual tuning of large language models with a small number of multilingual examples improves instruction-following capabilities across languages compared to monolingual tuning.
multilingual LLMs cross-lingual transfer instruction tuning multilinguality instruction-following monolingual tuning cross-lingual generalization