MELLA: Bridging Linguistic Capability and Cultural Groundedness for
  Low-Resource Language MLLMs
Multimodal Large Language Models (MLLMs) have shown remarkable performance in
high-resource languages. However, their effectiveness diminishes significantly
in the contexts of low-resource languages. Current multilingual enhancement
methods are often limited to text modality or rely solely on machine
translation. While such approaches help models acquire basic linguistic
capabilities and produce "thin descriptions", they neglect the importance of
multimodal informativeness and cultural groundedness, both of which are crucial
for serving low-resource language users effectively. To bridge this gap, in
this study, we identify two significant objectives for a truly effective MLLM
in low-resource language settings, namely 1) linguistic capability and 2)
cultural groundedness, placing special emphasis on cultural awareness. To
achieve these dual objectives, we propose a dual-source strategy that guides
the collection of data tailored to each goal, sourcing native web alt-text for
culture and MLLM-generated captions for linguistics. As a concrete
implementation, we introduce MELLA, a multimodal, multilingual dataset.
Experiment results show that after fine-tuning on MELLA, there is a general
performance improvement for the eight languages on various MLLM backbones, with
models producing "thick descriptions". We verify that the performance gains are
from both cultural knowledge enhancement and linguistic capability enhancement.
Our dataset can be found at https://opendatalab.com/applyMultilingualCorpus.
68994b7bf022d141f5d434be
Yufei Gao
68994b7bf022d141f5d434bf
Jiaying Fei
68994b7bf022d141f5d434c0
Nuo Chen
68994b7bf022d141f5d434c1
Ruirui Chen
68994b7bf022d141f5d434c2
Guohang Yan
68994b7bf022d141f5d434c3
Yunshi Lan
68994b7bf022d141f5d434c4
Botian Shi
MELLA: Bridging Linguistic Capability and Cultural Groundedness for
  Low-Resource Language MLLMs
Multimodal Large Language Models (MLLMs) have shown remarkable performance in
high-resource languages. However, their effectiveness diminishes significantly
in the contexts of low-resource languages. Current multilingual enhancement
methods are often limited to text modality or rely solely on machine
translation. While such approaches help models acquire basic linguistic
capabilities and produce "thin descriptions", they neglect the importance of
multimodal informativeness and cultural groundedness, both of which are crucial
for serving low-resource language users effectively. To bridge this gap, in
this study, we identify two significant objectives for a truly effective MLLM
in low-resource language settings, namely 1) linguistic capability and 2)
cultural groundedness, placing special emphasis on cultural awareness. To
achieve these dual objectives, we propose a dual-source strategy that guides
the collection of data tailored to each goal, sourcing native web alt-text for
culture and MLLM-generated captions for linguistics. As a concrete
implementation, we introduce MELLA, a multimodal, multilingual dataset.
Experiment results show that after fine-tuning on MELLA, there is a general
performance improvement for the eight languages on various MLLM backbones, with
models producing "thick descriptions". We verify that the performance gains are
from both cultural knowledge enhancement and linguistic capability enhancement.
Our dataset can be found at https://opendatalab.com/applyMultilingualCorpus.
MELLA, a multimodal, multilingual dataset, enhances MLLMs in low-resource languages by improving linguistic capability and cultural groundedness through native web alt-text and MLLM-generated captions.
Multimodal Large Language Models MLLMs low-resource languages multilingual enhancement native web alt-text MLLM-generated captions cultural awareness thick descriptions