An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
Visual instruction tuning has recently shown encouraging progress with
open-source large multimodal models (LMM) such as LLaVA and MiniGPT-4. However,
most existing studies of open-source LMM are performed using models with 13B
parameters or smaller. In this paper we present an empirical study of scaling
LLaVA up to 33B and 65B/70B, and share our findings from our explorations in
image resolution, data mixing and parameter-efficient training methods such as
LoRA/QLoRA. These are evaluated by their impact on the multi-modal and language
capabilities when completing real-world tasks in the wild.
  We find that scaling LMM consistently enhances model performance and improves
language capabilities, and performance of LoRA/QLoRA tuning of LMM are
comparable to the performance of full-model fine-tuning. Additionally, the
study highlights the importance of higher image resolutions and mixing
multimodal-language data to improve LMM performance, and visual instruction
tuning can sometimes improve LMM's pure language capability. We hope that this
study makes state-of-the-art LMM research at a larger scale more accessible,
thus helping establish stronger baselines for future research. Code and
checkpoints will be made public.
6508fc9a3499309131a1501b
Yadong Lu
admin_assigned
2023-09-20T13:14:13.931Z
64f570ff96185818de7a1ccb
/avatars/5d4e0a3d21fabccc7c02c441f5786a88.svg
Yadong Lu
user
Adong17
6508fc9a3499309131a1501c
Chunyuan Li
admin_assigned
2023-09-20T13:14:20.971Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
6508fc9a3499309131a1501d
Haotian Liu
admin_assigned
2023-09-20T13:14:38.578Z
63898b61ec1f539adc0f4da2
https://cdn-avatars.huggingface.co/v1/production/uploads/1674248167280-63898b61ec1f539adc0f4da2.jpeg
Haotian Liu
user
liuhaotian
6508fc9a3499309131a1501e
Jianwei Yang
admin_assigned
2023-09-20T13:15:33.011Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
6508fc9a3499309131a1501f
Jianfeng Gao
admin_assigned
2023-09-20T13:15:39.494Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
6508fc9a3499309131a15020
Yelong Shen
admin_assigned
2023-09-20T13:15:45.392Z
6454c337a13edf669cd5d8ea
/avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
Yelong Shen
user
uuu6
An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
Visual instruction tuning has recently shown encouraging progress with
open-source large multimodal models (LMM) such as LLaVA and MiniGPT-4. However,
most existing studies of open-source LMM are performed using models with 13B
parameters or smaller. In this paper we present an empirical study of scaling
LLaVA up to 33B and 65B/70B, and share our findings from our explorations in
image resolution, data mixing and parameter-efficient training methods such as
LoRA/QLoRA. These are evaluated by their impact on the multi-modal and language
capabilities when completing real-world tasks in the wild.
  We find that scaling LMM consistently enhances model performance and improves
language capabilities, and performance of LoRA/QLoRA tuning of LMM are
comparable to the performance of full-model fine-tuning. Additionally, the
study highlights the importance of higher image resolutions and mixing
multimodal-language data to improve LMM performance, and visual instruction
tuning can sometimes improve LMM's pure language capability. We hope that this
study makes state-of-the-art LMM research at a larger scale more accessible,
thus helping establish stronger baselines for future research. Code and
checkpoints will be made public.
Scaling LLaVA to 33B and 65B/70B parameters enhances performance, with LoRA/QLoRA tuning comparable to full-model fine-tuning, and highlights the importance of higher image resolutions and multimodal data mixing.
LLaVA MiniGPT-4 parameter-efficient training methods LoRA QLoRA full-model fine-tuning image resolution data mixing multi-modal capabilities language capabilities real-world tasks