Awaker2.5-VL: Stably Scaling MLLMs with Parameter-Efficient Mixture of
  Experts
As the research of Multimodal Large Language Models (MLLMs) becomes popular,
an advancing MLLM model is typically required to handle various textual and
visual tasks (e.g., VQA, Detection, OCR, and ChartQA) simultaneously for
real-world applications. However, due to the significant differences in
representation and distribution among data from various tasks, simply mixing
data of all tasks together leads to the well-known``multi-task conflict" issue,
resulting in performance degradation across various tasks. To address this
issue, we propose Awaker2.5-VL, a Mixture of Experts~(MoE) architecture
suitable for MLLM, which acquires the multi-task capabilities through multiple
sparsely activated experts. To speed up the training and inference of
Awaker2.5-VL, each expert in our model is devised as a low-rank adaptation
(LoRA) structure. Extensive experiments on multiple latest benchmarks
demonstrate the effectiveness of Awaker2.5-VL. The code and model weight are
released in our Project Page: https://github.com/MetabrainAGI/Awaker.
673c25ff11b7efeeed69461d
Jinqiang Long
claimed_verified
2024-11-20T10:09:08.488Z
6225f8edb471983f28f8fc44
/avatars/00ca46b8a745540b2ec1cb135fe02194.svg
Long
user
seventeen
673c25ff11b7efeeed69461e
Yanqi Dai
admin_assigned
2024-11-19T15:23:30.039Z
66cde57cb1fe4c78fe3ab770
https://cdn-avatars.huggingface.co/v1/production/uploads/66cde57cb1fe4c78fe3ab770/0R1aA-f_XLjCfy1HwqZ-p.jpeg
Yanqi Dai
user
YanqiDai
673c25ff11b7efeeed69461f
Guoxing Yang
673c25ff11b7efeeed694620
Hongpeng Lin
673c25ff11b7efeeed694621
Nanyi Fei
673c25ff11b7efeeed694622
Yizhao Gao
admin_assigned
2024-11-19T15:23:56.495Z
661c96f48921f03a9dae04c3
/avatars/c486a45ffea7d1a8c72bd8512014b07e.svg
Yizhao Gao
user
LongMountain
673c25ff11b7efeeed694623
Zhiwu Lu
admin_assigned
2024-11-19T15:23:50.216Z
6351e3035b904878f1d36719
/avatars/a7b3a5faacaa483fab1a5b1b95bcc0a4.svg
Zhiwu Lu
user
zewlu
Awaker2.5-VL: Stably Scaling MLLMs with Parameter-Efficient Mixture of
  Experts
As the research of Multimodal Large Language Models (MLLMs) becomes popular,
an advancing MLLM model is typically required to handle various textual and
visual tasks (e.g., VQA, Detection, OCR, and ChartQA) simultaneously for
real-world applications. However, due to the significant differences in
representation and distribution among data from various tasks, simply mixing
data of all tasks together leads to the well-known``multi-task conflict" issue,
resulting in performance degradation across various tasks. To address this
issue, we propose Awaker2.5-VL, a Mixture of Experts~(MoE) architecture
suitable for MLLM, which acquires the multi-task capabilities through multiple
sparsely activated experts. To speed up the training and inference of
Awaker2.5-VL, each expert in our model is devised as a low-rank adaptation
(LoRA) structure. Extensive experiments on multiple latest benchmarks
demonstrate the effectiveness of Awaker2.5-VL. The code and model weight are
released in our Project Page: https://github.com/MetabrainAGI/Awaker.
Awaker2.5-VL uses a Mixture of Experts architecture with low-rank adaptation to improve performance in various multimodal tasks.
Multimodal Large Language Models MLLMs VQA Detection OCR ChartQA multi-task conflict Mixture of Experts MoE low-rank adaptation LoRA