Imp: Highly Capable Large Multimodal Models for Mobile Devices
By harnessing the capabilities of large language models (LLMs), recent large
multimodal models (LMMs) have shown remarkable versatility in open-world
multimodal understanding. Nevertheless, they are usually parameter-heavy and
computation-intensive, thus hindering their applicability in
resource-constrained scenarios. To this end, several lightweight LMMs have been
proposed successively to maximize the capabilities under constrained scale
(e.g., 3B). Despite the encouraging results achieved by these methods, most of
them only focus on one or two aspects of the design space, and the key design
choices that influence model capability have not yet been thoroughly
investigated. In this paper, we conduct a systematic study for lightweight LMMs
from the aspects of model architecture, training strategy, and training data.
Based on our findings, we obtain Imp -- a family of highly capable LMMs at the
2B-4B scales. Notably, our Imp-3B model steadily outperforms all the existing
lightweight LMMs of similar size, and even surpasses the state-of-the-art LMMs
at the 13B scale. With low-bit quantization and resolution reduction
techniques, our Imp model can be deployed on a Qualcomm Snapdragon 8Gen3 mobile
chip with a high inference speed of about 13 tokens/s.
664c112f877f8871f445dbbc
Zhenwei Shao
admin_assigned
2024-05-21T08:37:43.401Z
62d7ce9243df7719860aee18
/avatars/02d06fe43256a28606c0bb63fed1a441.svg
Zhenwei Shao
user
Zenwill
664c112f877f8871f445dbbd
Zhou Yu
664c112f877f8871f445dbbe
Jun Yu
664c112f877f8871f445dbbf
Xuecheng Ouyang
664c112f877f8871f445dbc0
Lihao Zheng
admin_assigned
2024-05-21T08:38:41.554Z
635dcf0b1e2d8c6673e1255a
/avatars/174b56f7fe6e0c04de9cf4e387912214.svg
lihao zheng
user
NEE2641
664c112f877f8871f445dbc1
Zhenbiao Gai
664c112f877f8871f445dbc2
Mingyang Wang
664c112f877f8871f445dbc3
Jiajun Ding
admin_assigned
2024-05-21T08:39:05.217Z
64c336527f9b6ae10e114d4d
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/hlvIpQa4fACIeH9w1hjOZ.jpeg
Dingjiajun
user
Kissu4KillU
Imp: Highly Capable Large Multimodal Models for Mobile Devices
By harnessing the capabilities of large language models (LLMs), recent large
multimodal models (LMMs) have shown remarkable versatility in open-world
multimodal understanding. Nevertheless, they are usually parameter-heavy and
computation-intensive, thus hindering their applicability in
resource-constrained scenarios. To this end, several lightweight LMMs have been
proposed successively to maximize the capabilities under constrained scale
(e.g., 3B). Despite the encouraging results achieved by these methods, most of
them only focus on one or two aspects of the design space, and the key design
choices that influence model capability have not yet been thoroughly
investigated. In this paper, we conduct a systematic study for lightweight LMMs
from the aspects of model architecture, training strategy, and training data.
Based on our findings, we obtain Imp -- a family of highly capable LMMs at the
2B-4B scales. Notably, our Imp-3B model steadily outperforms all the existing
lightweight LMMs of similar size, and even surpasses the state-of-the-art LMMs
at the 13B scale. With low-bit quantization and resolution reduction
techniques, our Imp model can be deployed on a Qualcomm Snapdragon 8Gen3 mobile
chip with a high inference speed of about 13 tokens/s.
A systematic study of lightweight large multimodal models (LMMs) led to the development of the Imp family, which achieves superior performance compared to larger models and is capable of high-speed inference on mobile devices.
large language models large multimodal models parameter-heavy computation-intensive model architecture training strategy training data low-bit quantization resolution reduction Qualcomm Snapdragon 8Gen3