PALO: A Polyglot Large Multimodal Model for 5B People
In pursuit of more inclusive Vision-Language Models (VLMs), this study
introduces a Large Multilingual Multimodal Model called Palo.
Palo offers visual reasoning capabilities in 10 major languages,
including English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian,
Urdu, and Japanese, that span a total of sim5B people (65\% of the world
population). Our approach involves a semi-automated translation approach to
adapt the multimodal instruction dataset from English to the target languages
using a fine-tuned Large Language Model, thereby ensuring high linguistic
fidelity while allowing scalability due to minimal manual effort. The
incorporation of diverse instruction sets helps us boost overall performance
across multiple languages especially those that are underrepresented like
Hindi, Arabic, Bengali, and Urdu. The resulting models are trained across three
scales (1.7B, 7B and 13B parameters) to show the generalization and scalability
where we observe substantial improvements compared to strong baselines. We also
propose the first multilingual multimodal benchmark for the forthcoming
approaches to evaluate their vision-language reasoning capabilities across
languages. Code: https://github.com/mbzuai-oryx/PALO.
65d813fa0be3fc13a9de8fab
Muhammad Maaz
claimed_verified
2024-02-26T09:25:48.560Z
64807585856901b0edb8d68b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/kmCo6brXhQ7SyxVx3lpsx.jpeg
Muhammad Maaz
user
mmaaz60
65d813fa0be3fc13a9de8fac
Hanoona Rasheed
admin_assigned
2024-02-23T12:48:18.019Z
64636b2551fa6e6306046293
https://cdn-avatars.huggingface.co/v1/production/uploads/64636b2551fa6e6306046293/Uuz6z2MZb_LKLGM8uxF9s.jpeg
Hanoona Rasheed
user
Hanoona
65d813fa0be3fc13a9de8fad
Abdelrahman Shaker
65d813fa0be3fc13a9de8fae
Salman Khan
65d813fa0be3fc13a9de8faf
Hisham Cholakal
65d813fa0be3fc13a9de8fb0
Rao M. Anwer
65d813fa0be3fc13a9de8fb1
Tim Baldwin
claimed_verified
2024-02-23T13:02:29.589Z
65d895d8d435dce78392065f
/avatars/68c7185d3dd1adf78927b91849978d62.svg
Tim Baldwin
user
tbaldwin
65d813fa0be3fc13a9de8fb2
Michael Felsberg
admin_assigned
2024-02-23T12:49:10.186Z
63d38a1a9e1b81818c816905
/avatars/a2160bb9fbb8a8eec129b8eab559294b.svg
Michael Felsberg
user
CVL-LiU
65d813fa0be3fc13a9de8fb3
Fahad S. Khan
PALO: A Polyglot Large Multimodal Model for 5B People
In pursuit of more inclusive Vision-Language Models (VLMs), this study
introduces a Large Multilingual Multimodal Model called Palo.
Palo offers visual reasoning capabilities in 10 major languages,
including English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian,
Urdu, and Japanese, that span a total of sim5B people (65\% of the world
population). Our approach involves a semi-automated translation approach to
adapt the multimodal instruction dataset from English to the target languages
using a fine-tuned Large Language Model, thereby ensuring high linguistic
fidelity while allowing scalability due to minimal manual effort. The
incorporation of diverse instruction sets helps us boost overall performance
across multiple languages especially those that are underrepresented like
Hindi, Arabic, Bengali, and Urdu. The resulting models are trained across three
scales (1.7B, 7B and 13B parameters) to show the generalization and scalability
where we observe substantial improvements compared to strong baselines. We also
propose the first multilingual multimodal benchmark for the forthcoming
approaches to evaluate their vision-language reasoning capabilities across
languages. Code: https://github.com/mbzuai-oryx/PALO.
A multilingual multimodal model called Palo enhances visual reasoning across 10 languages using a semi-automated translation approach and achieves substantial performance improvements across multiple scales.
Large Multilingual Multimodal Model visual reasoning Palo semi-automated translation fine-tuned Large Language Model multilingual multimodal benchmark