TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones
In the era of advanced multimodel learning, multimodal large language models
(MLLMs) such as GPT-4V have made remarkable strides towards bridging language
and visual elements. However, the closed-source nature and considerable
computational demand present notable challenges for universal usage and
modifications. This is where open-source MLLMs like LLaVA and MiniGPT-4 come
in, presenting groundbreaking achievements across tasks. Despite these
accomplishments, computational efficiency remains an unresolved issue, as these
models, like LLaVA-v1.5-13B, require substantial resources. Addressing these
issues, we introduce TinyGPT-V, a new-wave model marrying impressive
performance with commonplace computational capacity. It stands out by requiring
merely a 24G GPU for training and an 8G GPU or CPU for inference. Built upon
Phi-2, TinyGPT-V couples an effective language backbone with pre-trained vision
modules from BLIP-2 or CLIP. TinyGPT-V's 2.8B parameters can undergo a unique
quantisation process, suitable for local deployment and inference tasks on 8G
various devices. Our work fosters further developments for designing
cost-effective, efficient, and high-performing MLLMs, expanding their
applicability in a broad array of real-world scenarios. Furthermore this paper
proposed a new paradigm of Multimodal Large Language Model via small backbones.
Our code and training weights are placed at:
https://github.com/DLYuanGod/TinyGPT-V and
https://huggingface.co/Tyrannosaurus/TinyGPT-V respectively.
658e28fdf7291078f975c8f7
Zhengqing Yuan
extracted_confirmed
2023-12-29T02:04:23.827Z
64574d8e182c64e989846ba2
/avatars/db4bc496a745e1d7de48215b30f6fd3e.svg
Tyrannosaurus
user
Tyrannosaurus
658e28fdf7291078f975c8f8
Zhaoxu Li
claimed_verified
2024-01-04T12:41:23.104Z
64c3c7fd4399efa2fd6a348a
/avatars/325dccd3ce65205ccc464ada74764683.svg
LI ZHAOXU
user
llizhx
658e28fdf7291078f975c8f9
Lichao Sun
claimed_verified
2024-03-21T08:29:18.210Z
65a52766215aabac489e3468
/avatars/fe05e22cd7e12e961296426434e17c76.svg
Lichao Sun
user
sunlichao137
TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones
In the era of advanced multimodel learning, multimodal large language models
(MLLMs) such as GPT-4V have made remarkable strides towards bridging language
and visual elements. However, the closed-source nature and considerable
computational demand present notable challenges for universal usage and
modifications. This is where open-source MLLMs like LLaVA and MiniGPT-4 come
in, presenting groundbreaking achievements across tasks. Despite these
accomplishments, computational efficiency remains an unresolved issue, as these
models, like LLaVA-v1.5-13B, require substantial resources. Addressing these
issues, we introduce TinyGPT-V, a new-wave model marrying impressive
performance with commonplace computational capacity. It stands out by requiring
merely a 24G GPU for training and an 8G GPU or CPU for inference. Built upon
Phi-2, TinyGPT-V couples an effective language backbone with pre-trained vision
modules from BLIP-2 or CLIP. TinyGPT-V's 2.8B parameters can undergo a unique
quantisation process, suitable for local deployment and inference tasks on 8G
various devices. Our work fosters further developments for designing
cost-effective, efficient, and high-performing MLLMs, expanding their
applicability in a broad array of real-world scenarios. Furthermore this paper
proposed a new paradigm of Multimodal Large Language Model via small backbones.
Our code and training weights are placed at:
https://github.com/DLYuanGod/TinyGPT-V and
https://huggingface.co/Tyrannosaurus/TinyGPT-V respectively.
TinyGPT-V, built on Phi-2 with vision modules from BLIP-2 or CLIP, offers high performance with low computational requirements, enabling efficient multimodal large language modeling.
multimodal large language models MLLMs GPT-4V LLaVA MiniGPT-4 computational efficiency GPU inference Phi-2 BLIP-2 CLIP quantisation small backbones