Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large
  Language Models
Despite remarkable progress, existing multimodal large language models
(MLLMs) are still inferior in granular visual recognition. Contrary to previous
works, we study this problem from the perspective of image resolution, and
reveal that a combination of low- and high-resolution visual features can
effectively mitigate this shortcoming. Based on this observation, we propose a
novel and efficient method for MLLMs, termed Mixture-of-Resolution Adaptation
(MRA). In particular, MRA adopts two visual pathways for images with different
resolutions, where high-resolution visual information is embedded into the
low-resolution pathway via the novel mixture-of-resolution adapters
(MR-Adapters). This design also greatly reduces the input sequence length of
MLLMs. To validate MRA, we apply it to a recent MLLM called LLaVA, and term the
new model LLaVA-HR. We conduct extensive experiments on 11 vision-language (VL)
tasks, which show that LLaVA-HR outperforms existing MLLMs on 8 VL tasks, e.g.,
+9.4% on TextVQA. More importantly, both training and inference of LLaVA-HR
remain efficient with MRA, e.g., 20 training hours and 3times inference
speed than LLaVA-1.5. Source codes are released at:
https://github.com/luogen1996/LLaVA-HR.
65e7debcb87e5297e4670304
Gen Luo
65e7debcb87e5297e4670305
Yiyi Zhou
65e7debcb87e5297e4670306
Yuxin Zhang
65e7debcb87e5297e4670307
Xiawu Zheng
65e7debcb87e5297e4670308
Xiaoshuai Sun
65e7debcb87e5297e4670309
Rongrong Ji
Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large
  Language Models
Despite remarkable progress, existing multimodal large language models
(MLLMs) are still inferior in granular visual recognition. Contrary to previous
works, we study this problem from the perspective of image resolution, and
reveal that a combination of low- and high-resolution visual features can
effectively mitigate this shortcoming. Based on this observation, we propose a
novel and efficient method for MLLMs, termed Mixture-of-Resolution Adaptation
(MRA). In particular, MRA adopts two visual pathways for images with different
resolutions, where high-resolution visual information is embedded into the
low-resolution pathway via the novel mixture-of-resolution adapters
(MR-Adapters). This design also greatly reduces the input sequence length of
MLLMs. To validate MRA, we apply it to a recent MLLM called LLaVA, and term the
new model LLaVA-HR. We conduct extensive experiments on 11 vision-language (VL)
tasks, which show that LLaVA-HR outperforms existing MLLMs on 8 VL tasks, e.g.,
+9.4% on TextVQA. More importantly, both training and inference of LLaVA-HR
remain efficient with MRA, e.g., 20 training hours and 3times inference
speed than LLaVA-1.5. Source codes are released at:
https://github.com/luogen1996/LLaVA-HR.
A novel method called Mixture-of-Resolution Adaptation (MRA) enhances multimodal large language models by integrating low- and high-resolution visual features, improving visual recognition and efficiency.
Multimodal large language models MLLMs Mixture-of-Resolution Adaptation MRA mixture-of-resolution adapters MR-Adapters image resolution visual pathways high-resolution visual information vision-language tasks VL tasks LLaVA LLaVA-HR