Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages
Despite recent advances in multimodal large language models (MLLMs), their
development has predominantly focused on English- and western-centric datasets
and tasks, leaving most of the world's languages and diverse cultural contexts
underrepresented. This paper introduces Pangea, a multilingual multimodal LLM
trained on PangeaIns, a diverse 6M instruction dataset spanning 39 languages.
PangeaIns features: 1) high-quality English instructions, 2) carefully
machine-translated instructions, and 3) culturally relevant multimodal tasks to
ensure cross-cultural coverage. To rigorously assess models' capabilities, we
introduce PangeaBench, a holistic evaluation suite encompassing 14 datasets
covering 47 languages. Results show that Pangea significantly outperforms
existing open-source models in multilingual settings and diverse cultural
contexts. Ablation studies further reveal the importance of English data
proportions, language popularity, and the number of multimodal training samples
on overall performance. We fully open-source our data, code, and trained
checkpoints, to facilitate the development of inclusive and robust multilingual
MLLMs, promoting equity and accessibility across a broader linguistic and
cultural spectrum.
671718ed63c851e167b775c5
Xiang Yue
claimed_verified
2024-10-22T07:57:23.127Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
671718ed63c851e167b775c6
Yueqi Song
admin_assigned
2024-10-22T09:17:30.353Z
63e7bf7be02ee67e8e53f78d
https://cdn-avatars.huggingface.co/v1/production/uploads/63e7bf7be02ee67e8e53f78d/z9j1PWrurEyIA2JXzpz0i.png
Yueqi Song
user
yueqis
671718ed63c851e167b775c7
Akari Asai
admin_assigned
2024-10-22T09:17:24.248Z
6266e0cb7a1f5a1562c4e86e
https://cdn-avatars.huggingface.co/v1/production/uploads/6266e0cb7a1f5a1562c4e86e/kkJZPssa76mdRPnKH-q3e.jpeg
Akari Asai
user
akariasai
671718ed63c851e167b775c8
Seungone Kim
claimed_verified
2024-10-22T07:57:21.588Z
6469949654873f0043b09c22
https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
Seungone Kim
user
seungone
671718ed63c851e167b775c9
Jean de Dieu Nyandwi
claimed_verified
2024-10-22T09:05:59.579Z
62989cd743e990340bead77c
https://cdn-avatars.huggingface.co/v1/production/uploads/1654168730944-noauth.png
Jean de Dieu Nyandwi
user
Nyandwi
671718ed63c851e167b775ca
Simran Khanuja
admin_assigned
2024-10-22T09:17:15.486Z
63a4d079658851481f7e4394
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4d079658851481f7e4394/yptj_hJiaqHvEaqxLs0jT.jpeg
Simran Khanuja
user
skhanuja
671718ed63c851e167b775cb
Anjali Kantharuban
admin_assigned
2024-10-22T09:16:49.370Z
634e6d7ea51d5df8c2da699d
https://cdn-avatars.huggingface.co/v1/production/uploads/1666084174110-noauth.jpeg
Anjali Kantharuban
user
AnjaliRuban
671718ed63c851e167b775cc
Lintang Sutawika
admin_assigned
2024-10-22T09:17:08.786Z
5f666a6b5e78cc6b0ed319fc
https://cdn-avatars.huggingface.co/v1/production/uploads/1635412047803-5f666a6b5e78cc6b0ed319fc.jpeg
Lintang Sutawika
user
lintang
671718ed63c851e167b775cd
Sathyanarayanan Ramamoorthy
admin_assigned
2024-10-22T09:17:02.898Z
643b87db0e5495afdeff8de5
/avatars/5c0bca81a52ab6b7beb7a52d82e53a10.svg
Sathyanarayanan Ramamoorthy
user
sathyaram
671718ed63c851e167b775ce
Graham Neubig
admin_assigned
2024-10-22T09:16:57.085Z
60de14638bedd2315529d43f
https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
Graham Neubig
user
gneubig
Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages
Despite recent advances in multimodal large language models (MLLMs), their
development has predominantly focused on English- and western-centric datasets
and tasks, leaving most of the world's languages and diverse cultural contexts
underrepresented. This paper introduces Pangea, a multilingual multimodal LLM
trained on PangeaIns, a diverse 6M instruction dataset spanning 39 languages.
PangeaIns features: 1) high-quality English instructions, 2) carefully
machine-translated instructions, and 3) culturally relevant multimodal tasks to
ensure cross-cultural coverage. To rigorously assess models' capabilities, we
introduce PangeaBench, a holistic evaluation suite encompassing 14 datasets
covering 47 languages. Results show that Pangea significantly outperforms
existing open-source models in multilingual settings and diverse cultural
contexts. Ablation studies further reveal the importance of English data
proportions, language popularity, and the number of multimodal training samples
on overall performance. We fully open-source our data, code, and trained
checkpoints, to facilitate the development of inclusive and robust multilingual
MLLMs, promoting equity and accessibility across a broader linguistic and
cultural spectrum.
Pangea, a multilingual multimodal LLM, is trained on a diverse 6M instruction dataset across 39 languages and evaluated using PangeaBench, demonstrating superior performance in multilingual and cross-cultural settings.
multimodal large language models (MLLMs) Pangea PangeaIns PangeaBench multilingual multimodal evaluation suite ablation studies