TinyLLaVA: A Framework of Small-scale Large Multimodal Models
We present the TinyLLaVA framework that provides a unified perspective in
designing and analyzing the small-scale Large Multimodal Models (LMMs). We
empirically study the effects of different vision encoders, connection modules,
language models, training data and training recipes. Our extensive experiments
showed that better quality of data combined with better training recipes,
smaller LMMs can consistently achieve on-par performances compared to bigger
LMMs. Under our framework, we train a family of small-scale LMMs. Our best
model, TinyLLaVA-3.1B, achieves better overall performance against existing 7B
models such as LLaVA-1.5 and Qwen-VL. We hope our findings can serve as
baselines for future research in terms of data scaling, training setups and
model selections. Our model weights and codes will be made public.
65d810553381a5be2779b3d3
Baichuan Zhou
claimed_verified
2024-02-23T12:25:13.725Z
646325085897b675c65aea0f
/avatars/28ce7388f9318b49bdd0a5594c0f6732.svg
Baichuan Zhou
user
bczhou
65d810553381a5be2779b3d4
Ying Hu
65d810553381a5be2779b3d5
Xi Weng
65d810553381a5be2779b3d6
Junlong Jia
65d810553381a5be2779b3d7
Jie Luo
65d810553381a5be2779b3d8
Xien Liu
65d810553381a5be2779b3d9
Ji Wu
65d810553381a5be2779b3da
Lei Huang
TinyLLaVA: A Framework of Small-scale Large Multimodal Models
We present the TinyLLaVA framework that provides a unified perspective in
designing and analyzing the small-scale Large Multimodal Models (LMMs). We
empirically study the effects of different vision encoders, connection modules,
language models, training data and training recipes. Our extensive experiments
showed that better quality of data combined with better training recipes,
smaller LMMs can consistently achieve on-par performances compared to bigger
LMMs. Under our framework, we train a family of small-scale LMMs. Our best
model, TinyLLaVA-3.1B, achieves better overall performance against existing 7B
models such as LLaVA-1.5 and Qwen-VL. We hope our findings can serve as
baselines for future research in terms of data scaling, training setups and
model selections. Our model weights and codes will be made public.
TinyLLaVA framework demonstrates that small-scale Large Multimodal Models can achieve comparable performance to larger models through improved data quality and training recipes.
Small-scale Large Multimodal Models vision encoders connection modules language models training data training recipes TinyLLaVA-3.1B LLaVA-1.5 Qwen-VL data scaling model selections