Centurio: On Drivers of Multilingual Ability of Large Vision-Language
  Model
Most Large Vision-Language Models (LVLMs) to date are trained predominantly
on English data, which makes them struggle to understand non-English input and
fail to generate output in the desired target language. Existing efforts
mitigate these issues by adding multilingual training data, but do so in a
largely ad-hoc manner, lacking insight into how different training mixes tip
the scale for different groups of languages. In this work, we present a
comprehensive investigation into the training strategies for massively
multilingual LVLMs. First, we conduct a series of multi-stage experiments
spanning 13 downstream vision-language tasks and 43 languages, systematically
examining: (1) the number of training languages that can be included without
degrading English performance and (2) optimal language distributions of
pre-training as well as (3) instruction-tuning data. Further, we (4)
investigate how to improve multilingual text-in-image understanding, and
introduce a new benchmark for the task. Surprisingly, our analysis reveals that
one can (i) include as many as 100 training languages simultaneously (ii) with
as little as 25-50\% of non-English data, to greatly improve multilingual
performance while retaining strong English performance. We further find that
(iii) including non-English OCR data in pre-training and instruction-tuning is
paramount for improving multilingual text-in-image understanding. Finally, we
put all our findings together and train Centurio, a 100-language LVLM, offering
state-of-the-art performance in an evaluation covering 14 tasks and 56
languages.
6780ce57c759e70936ba6da5
Gregor Geigle
admin_assigned
2025-01-10T13:35:02.224Z
613f291fe5a812eff913808e
https://cdn-avatars.huggingface.co/v1/production/uploads/613f291fe5a812eff913808e/aQUoRv6tllC2j-jg6meUt.jpeg
Gregor Geigle
user
Gregor
6780ce57c759e70936ba6da6
Florian Schneider
claimed_verified
2025-01-12T19:57:35.865Z
62dfd54798815401141c47fe
https://cdn-avatars.huggingface.co/v1/production/uploads/62dfd54798815401141c47fe/ct2OA_K0Wwpshy8DCswxy.png
Flo Schneider
user
floschne
6780ce57c759e70936ba6da7
Carolin Holtermann
6780ce57c759e70936ba6da8
Chris Biemann
6780ce57c759e70936ba6da9
Radu Timofte
6780ce57c759e70936ba6daa
Anne Lauscher
claimed_verified
2025-01-12T19:57:33.626Z
626c02e7703f3b27dd590896
https://cdn-avatars.huggingface.co/v1/production/uploads/1654503075060-626c02e7703f3b27dd590896.jpeg
Anne Lauscher
user
anlausch
6780ce57c759e70936ba6dab
Goran Glavaš
Centurio: On Drivers of Multilingual Ability of Large Vision-Language
  Model
Most Large Vision-Language Models (LVLMs) to date are trained predominantly
on English data, which makes them struggle to understand non-English input and
fail to generate output in the desired target language. Existing efforts
mitigate these issues by adding multilingual training data, but do so in a
largely ad-hoc manner, lacking insight into how different training mixes tip
the scale for different groups of languages. In this work, we present a
comprehensive investigation into the training strategies for massively
multilingual LVLMs. First, we conduct a series of multi-stage experiments
spanning 13 downstream vision-language tasks and 43 languages, systematically
examining: (1) the number of training languages that can be included without
degrading English performance and (2) optimal language distributions of
pre-training as well as (3) instruction-tuning data. Further, we (4)
investigate how to improve multilingual text-in-image understanding, and
introduce a new benchmark for the task. Surprisingly, our analysis reveals that
one can (i) include as many as 100 training languages simultaneously (ii) with
as little as 25-50\% of non-English data, to greatly improve multilingual
performance while retaining strong English performance. We further find that
(iii) including non-English OCR data in pre-training and instruction-tuning is
paramount for improving multilingual text-in-image understanding. Finally, we
put all our findings together and train Centurio, a 100-language LVLM, offering
state-of-the-art performance in an evaluation covering 14 tasks and 56
languages.
Large Vision-Language Models (LVLMs) multilingual training data downstream vision-language tasks English performance multilingual text-in-image understanding non-English OCR data pre-training instruction-tuning adiabetic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adi