Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models
Seeing clearly with high resolution is a foundation of Large Multimodal
Models (LMMs), which has been proven to be vital for visual perception and
reasoning. Existing works usually employ a straightforward resolution upscaling
method, where the image consists of global and local branches, with the latter
being the sliced image patches but resized to the same resolution as the
former. This means that higher resolution requires more local patches,
resulting in exorbitant computational expenses, and meanwhile, the dominance of
local image tokens may diminish the global context. In this paper, we dive into
the problems and propose a new framework as well as an elaborate optimization
strategy. Specifically, we extract contextual information from the global view
using a mixture of adapters, based on the observation that different adapters
excel at different tasks. With regard to local patches, learnable query
embeddings are introduced to reduce image tokens, the most important tokens
accounting for the user question will be further selected by a similarity-based
selector. Our empirical results demonstrate a `less is more' pattern, where
utilizing fewer but more informative local image tokens leads to
improved performance. Besides, a significant challenge lies in the training
strategy, as simultaneous end-to-end training of the global mining block and
local compression block does not yield optimal results. We thus advocate for an
alternating training way, ensuring balanced learning between global and local
aspects. Finally, we also introduce a challenging dataset with high
requirements for image detail, enhancing the training of the local compression
layer. The proposed method, termed LMM with Sophisticated Tasks, Local image
compression, and Mixture of global Experts (SliME), achieves leading
performance across various benchmarks with only 2 million training data.
666a8722d70d5346028c92de
Yi-Fan Zhang
claimed_verified
2024-06-13T09:30:21.831Z
623d8ca4c29adf5ef6175615
https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
Yi-Fan Zhang
user
yifanzhang114
666a8722d70d5346028c92df
Qingsong Wen
666a8722d70d5346028c92e0
Chaoyou Fu
666a8722d70d5346028c92e1
Xue Wang
666a8722d70d5346028c92e2
Zhang Zhang
666a8722d70d5346028c92e3
Liang Wang
666a8722d70d5346028c92e4
Rong Jin
Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models
Seeing clearly with high resolution is a foundation of Large Multimodal
Models (LMMs), which has been proven to be vital for visual perception and
reasoning. Existing works usually employ a straightforward resolution upscaling
method, where the image consists of global and local branches, with the latter
being the sliced image patches but resized to the same resolution as the
former. This means that higher resolution requires more local patches,
resulting in exorbitant computational expenses, and meanwhile, the dominance of
local image tokens may diminish the global context. In this paper, we dive into
the problems and propose a new framework as well as an elaborate optimization
strategy. Specifically, we extract contextual information from the global view
using a mixture of adapters, based on the observation that different adapters
excel at different tasks. With regard to local patches, learnable query
embeddings are introduced to reduce image tokens, the most important tokens
accounting for the user question will be further selected by a similarity-based
selector. Our empirical results demonstrate a `less is more' pattern, where
utilizing fewer but more informative local image tokens leads to
improved performance. Besides, a significant challenge lies in the training
strategy, as simultaneous end-to-end training of the global mining block and
local compression block does not yield optimal results. We thus advocate for an
alternating training way, ensuring balanced learning between global and local
aspects. Finally, we also introduce a challenging dataset with high
requirements for image detail, enhancing the training of the local compression
layer. The proposed method, termed LMM with Sophisticated Tasks, Local image
compression, and Mixture of global Experts (SliME), achieves leading
performance across various benchmarks with only 2 million training data.
A new framework for Large Multimodal Models (LMMs) addresses computational and context challenges by using a mix of global adapters and local compression, achieving high performance with efficient training.
Large Multimodal Models high resolution resolution upscaling global view adapters local patches learnable query embeddings similarity-based selector less is more end-to-end training alternating training local compression global mining block local aspects SliME