Dense Connector for MLLMs
Do we fully leverage the potential of visual encoder in Multimodal Large
Language Models (MLLMs)? The recent outstanding performance of MLLMs in
multimodal understanding has garnered broad attention from both academia and
industry. In the current MLLM rat race, the focus seems to be predominantly on
the linguistic side. We witness the rise of larger and higher-quality
instruction datasets, as well as the involvement of larger-sized LLMs. Yet,
scant attention has been directed towards the visual signals utilized by MLLMs,
often assumed to be the final high-level features extracted by a frozen visual
encoder. In this paper, we introduce the Dense Connector - a simple, effective,
and plug-and-play vision-language connector that significantly enhances
existing MLLMs by leveraging multi-layer visual features, with minimal
additional computational overhead. Furthermore, our model, trained solely on
images, showcases remarkable zero-shot capabilities in video understanding as
well. Experimental results across various vision encoders, image resolutions,
training dataset scales, varying sizes of LLMs (2.7B->70B), and diverse
architectures of MLLMs (e.g., LLaVA and Mini-Gemini) validate the versatility
and scalability of our approach, achieving state-of-the-art performance on
across 19 image and video benchmarks. We hope that this work will provide
valuable experience and serve as a basic module for future MLLM development.
6650187ca120d855a7f00a10
Huanjin Yao
admin_assigned
2024-05-24T10:59:37.410Z
6590e03454f8826173ed5ee6
/avatars/f5e59d3e58c28a99f2ff39267ca51cdb.svg
Huanjin Yao
user
HuanjinYao
6650187ca120d855a7f00a11
Wenhao Wu
admin_assigned
2024-05-24T10:59:44.675Z
64b66ddbd5a02e636b577dcc
https://cdn-avatars.huggingface.co/v1/production/uploads/64b66ddbd5a02e636b577dcc/61YPkyef3m3k5dpVI1d86.jpeg
Wenhao Wu
user
whwu-me
6650187ca120d855a7f00a12
Taojiannan Yang
6650187ca120d855a7f00a13
YuXin Song
admin_assigned
2024-05-24T11:00:03.632Z
63ad98c142fd3b8dbae78f51
/avatars/d7a96f0ad47a729027757b850b5e9712.svg
Yuxin Song
user
syxbb
6650187ca120d855a7f00a14
Mengxi Zhang
admin_assigned
2024-05-24T11:00:09.604Z
65f2a7267d5cdbce8882548b
/avatars/f0244fa4e71403d05ea8bfd25986a02b.svg
mengxizhang
user
mengxizhang
6650187ca120d855a7f00a15
Haocheng Feng
6650187ca120d855a7f00a16
Yifan Sun
6650187ca120d855a7f00a17
Zhiheng Li
admin_assigned
2024-05-24T11:01:22.958Z
60f86f2085dbbb185d2e9a5b
/avatars/52b0a8471115a8aa400e315224530e8a.svg
Zhiheng Li
user
hubertlee915
6650187ca120d855a7f00a18
Wanli Ouyang
6650187ca120d855a7f00a19
Jingdong Wang
Dense Connector for MLLMs
Do we fully leverage the potential of visual encoder in Multimodal Large
Language Models (MLLMs)? The recent outstanding performance of MLLMs in
multimodal understanding has garnered broad attention from both academia and
industry. In the current MLLM rat race, the focus seems to be predominantly on
the linguistic side. We witness the rise of larger and higher-quality
instruction datasets, as well as the involvement of larger-sized LLMs. Yet,
scant attention has been directed towards the visual signals utilized by MLLMs,
often assumed to be the final high-level features extracted by a frozen visual
encoder. In this paper, we introduce the Dense Connector - a simple, effective,
and plug-and-play vision-language connector that significantly enhances
existing MLLMs by leveraging multi-layer visual features, with minimal
additional computational overhead. Furthermore, our model, trained solely on
images, showcases remarkable zero-shot capabilities in video understanding as
well. Experimental results across various vision encoders, image resolutions,
training dataset scales, varying sizes of LLMs (2.7B->70B), and diverse
architectures of MLLMs (e.g., LLaVA and Mini-Gemini) validate the versatility
and scalability of our approach, achieving state-of-the-art performance on
across 19 image and video benchmarks. We hope that this work will provide
valuable experience and serve as a basic module for future MLLM development.
The Dense Connector enhances Multimodal Large Language Models by integrating multi-layer visual features, improving performance across image and video benchmarks.
Dense Connector vision-language connector multi-layer visual features Multimodal Large Language Models MLLMs zero-shot capabilities video understanding image and video benchmarks