MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
For Large Vision-Language Models (LVLMs), scaling the model can effectively
improve performance. However, expanding model parameters significantly
increases the training and inferring costs, as all model parameters are
activated for each token in the calculation. In this work, we propose a novel
training strategy MoE-tuning for LVLMs, which can constructing a sparse model
with an outrageous number of parameter but a constant computational cost, and
effectively addresses the performance degradation typically associated with
multi-modal learning and model sparsity. Furthermore, we present the MoE-LLaVA
framework, a MoE-based sparse LVLM architecture. This framework uniquely
activates only the top-k experts through routers during deployment, keeping the
remaining experts inactive. Our extensive experiments highlight the excellent
capabilities of MoE-LLaVA in visual understanding and its potential to reduce
hallucinations in model outputs. Remarkably, with just 3 billion sparsely
activated parameters, MoE-LLaVA demonstrates performance comparable to the
LLaVA-1.5-7B on various visual understanding datasets and even surpasses the
LLaVA-1.5-13B in object hallucination benchmarks. Through MoE-LLaVA, we aim to
establish a baseline for sparse LVLMs and provide valuable insights for future
research in developing more efficient and effective multi-modal learning
systems. Code is released at https://github.com/PKU-YuanGroup/MoE-LLaVA.
65b86b41e0bde92c1761c61a
Bin Lin
claimed_verified
2024-01-30T08:06:36.700Z
6367a8175bb06007ea099b8f
https://cdn-avatars.huggingface.co/v1/production/uploads/6367a8175bb06007ea099b8f/IjG7HyWyWRlVt_XwRbxRW.jpeg
linbin
user
LanguageBind
65b86b41e0bde92c1761c61b
Zhenyu Tang
admin_assigned
2024-01-30T09:15:08.478Z
63c09aa88d1175e3399e906e
/avatars/a1b8a89e83275f049f7bf75af9086d84.svg
Zhenyu Tang
user
royjames
65b86b41e0bde92c1761c61c
Yang Ye
65b86b41e0bde92c1761c61d
Jiaxi Cui
claimed_verified
2024-06-04T10:28:04.520Z
6389a80b4b547a364ea041ed
https://cdn-avatars.huggingface.co/v1/production/uploads/1669965806283-noauth.jpeg
Jessy Tsui
user
JessyTsu1
65b86b41e0bde92c1761c61e
Bin Zhu
65b86b41e0bde92c1761c61f
Peng Jin
claimed_verified
2024-10-17T11:31:19.940Z
651585ea18cfed0d30bee586
/avatars/579e468334102472d870875fe40302e6.svg
Peng Jin
user
Chat-UniVi
65b86b41e0bde92c1761c620
Junwu Zhang
admin_assigned
2024-01-30T09:16:15.836Z
64979385e486365ca6b3c030
/avatars/589344a9db41873d78452560cca3d234.svg
Junwu Zhang
user
junwuzhang
65b86b41e0bde92c1761c621
Munan Ning
admin_assigned
2024-01-30T09:16:22.621Z
62a9748708a7ea93ff6d48ab
/avatars/c6358f81e31f7d31e14899011ce67e09.svg
Munan Ning
user
MasonNing
65b86b41e0bde92c1761c622
Li Yuan
claimed_verified
2025-04-04T07:12:33.024Z
66135a5e50350afe76beebce
/avatars/370a4b83949355feb050c2cb0425c264.svg
yl2488
user
yl2488
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
For Large Vision-Language Models (LVLMs), scaling the model can effectively
improve performance. However, expanding model parameters significantly
increases the training and inferring costs, as all model parameters are
activated for each token in the calculation. In this work, we propose a novel
training strategy MoE-tuning for LVLMs, which can constructing a sparse model
with an outrageous number of parameter but a constant computational cost, and
effectively addresses the performance degradation typically associated with
multi-modal learning and model sparsity. Furthermore, we present the MoE-LLaVA
framework, a MoE-based sparse LVLM architecture. This framework uniquely
activates only the top-k experts through routers during deployment, keeping the
remaining experts inactive. Our extensive experiments highlight the excellent
capabilities of MoE-LLaVA in visual understanding and its potential to reduce
hallucinations in model outputs. Remarkably, with just 3 billion sparsely
activated parameters, MoE-LLaVA demonstrates performance comparable to the
LLaVA-1.5-7B on various visual understanding datasets and even surpasses the
LLaVA-1.5-13B in object hallucination benchmarks. Through MoE-LLaVA, we aim to
establish a baseline for sparse LVLMs and provide valuable insights for future
research in developing more efficient and effective multi-modal learning
systems. Code is released at https://github.com/PKU-YuanGroup/MoE-LLaVA.
MoE-LLaVA, a sparse Large Vision-Language Model architecture, improves performance and reduces hallucinations with fewer active parameters compared to larger models.
MoE-tuning LVLMs sparse model computational cost performance degradation multi-modal learning model sparsity MoE-LLaVA routers visual understanding object hallucination