Centurio: On Drivers of Multilingual Ability of Large Vision-Language
  Model
Most Large Vision-Language Models (LVLMs) to date are trained predominantly
on English data, which makes them struggle to understand non-English input and
fail to generate output in the desired target language. Existing efforts
mitigate these issues by adding multilingual training data, but do so in a
largely ad-hoc manner, lacking insight into how different training mixes tip
the scale for different groups of languages. In this work, we present a
comprehensive investigation into the training strategies for massively
multilingual LVLMs. First, we conduct a series of multi-stage experiments
spanning 13 downstream vision-language tasks and 43 languages, systematically
examining: (1) the number of training languages that can be included without
degrading English performance and (2) optimal language distributions of
pre-training as well as (3) instruction-tuning data. Further, we (4)
investigate how to improve multilingual text-in-image understanding, and
introduce a new benchmark for the task. Surprisingly, our analysis reveals that
one can (i) include as many as 100 training languages simultaneously (ii) with
as little as 25-50\% of non-English data, to greatly improve multilingual
performance while retaining strong English performance. We further find that
(iii) including non-English OCR data in pre-training and instruction-tuning is
paramount for improving multilingual text-in-image understanding. Finally, we
put all our findings together and train Centurio, a 100-language LVLM, offering
state-of-the-art performance in an evaluation covering 14 tasks and 56
languages.
6780ce57c759e70936ba6da5
Gregor Geigle
admin_assigned
2025-01-10T13:35:02.224Z
613f291fe5a812eff913808e
https://cdn-avatars.huggingface.co/v1/production/uploads/613f291fe5a812eff913808e/aQUoRv6tllC2j-jg6meUt.jpeg
Gregor Geigle
user
Gregor
6780ce57c759e70936ba6da6
Florian Schneider
claimed_verified
2025-01-12T19:57:35.865Z
62dfd54798815401141c47fe
https://cdn-avatars.huggingface.co/v1/production/uploads/62dfd54798815401141c47fe/ct2OA_K0Wwpshy8DCswxy.png
Flo Schneider
user
floschne
6780ce57c759e70936ba6da7
Carolin Holtermann
6780ce57c759e70936ba6da8
Chris Biemann
6780ce57c759e70936ba6da9
Radu Timofte
6780ce57c759e70936ba6daa
Anne Lauscher
claimed_verified
2025-01-12T19:57:33.626Z
626c02e7703f3b27dd590896
https://cdn-avatars.huggingface.co/v1/production/uploads/1654503075060-626c02e7703f3b27dd590896.jpeg
Anne Lauscher
user
anlausch
6780ce57c759e70936ba6dab
Goran Glavaš
Centurio: On Drivers of Multilingual Ability of Large Vision-Language
  Model
Most Large Vision-Language Models (LVLMs) to date are trained predominantly
on English data, which makes them struggle to understand non-English input and
fail to generate output in the desired target language. Existing efforts
mitigate these issues by adding multilingual training data, but do so in a
largely ad-hoc manner, lacking insight into how different training mixes tip
the scale for different groups of languages. In this work, we present a
comprehensive investigation into the training strategies for massively
multilingual LVLMs. First, we conduct a series of multi-stage experiments
spanning 13 downstream vision-language tasks and 43 languages, systematically
examining: (1) the number of training languages that can be included without
degrading English performance and (2) optimal language distributions of
pre-training as well as (3) instruction-tuning data. Further, we (4)
investigate how to improve multilingual text-in-image understanding, and
introduce a new benchmark for the task. Surprisingly, our analysis reveals that
one can (i) include as many as 100 training languages simultaneously (ii) with
as little as 25-50\% of non-English data, to greatly improve multilingual
performance while retaining strong English performance. We further find that
(iii) including non-English OCR data in pre-training and instruction-tuning is
paramount for improving multilingual text-in-image understanding. Finally, we
put all our findings together and train Centurio, a 100-language LVLM, offering
state-of-the-art performance in an evaluation covering 14 tasks and 56
languages.
A comprehensive investigation into training strategies for multilingual vision-language models reveals optimal ways to include multiple languages without degrading English performance and introduces a new benchmark for text-in-image understanding.
Large Vision-Language Models LVLMs multilingual training downstream vision-language tasks pre-training instruction-tuning multilingual text-in-image understanding benchmark