LLaVA-φ: Efficient Multi-Modal Assistant with Small Language Model
In this paper, we introduce LLaVA-phi (LLaVA-Phi), an efficient
multi-modal assistant that harnesses the power of the recently advanced small
language model, Phi-2, to facilitate multi-modal dialogues. LLaVA-Phi marks a
notable advancement in the realm of compact multi-modal models. It demonstrates
that even smaller language models, with as few as 2.7B parameters, can
effectively engage in intricate dialogues that integrate both textual and
visual elements, provided they are trained with high-quality corpora. Our model
delivers commendable performance on publicly available benchmarks that
encompass visual comprehension, reasoning, and knowledge-based perception.
Beyond its remarkable performance in multi-modal dialogue tasks, our model
opens new avenues for applications in time-sensitive environments and systems
that require real-time interaction, such as embodied agents. It highlights the
potential of smaller language models to achieve sophisticated levels of
understanding and interaction, while maintaining greater resource
efficiency.The project is available at {https://github.com/zhuyiche/llava-phi}.
65978c204dbb962b7c5126ba
Yichen Zhu
65978c204dbb962b7c5126bb
Minjie Zhu
65978c204dbb962b7c5126bc
Ning Liu
65978c204dbb962b7c5126bd
Zhicai Ou
65978c204dbb962b7c5126be
Xiaofeng Mou
65978c204dbb962b7c5126bf
Jian Tang
LLaVA-φ: Efficient Multi-Modal Assistant with Small Language Model
In this paper, we introduce LLaVA-phi (LLaVA-Phi), an efficient
multi-modal assistant that harnesses the power of the recently advanced small
language model, Phi-2, to facilitate multi-modal dialogues. LLaVA-Phi marks a
notable advancement in the realm of compact multi-modal models. It demonstrates
that even smaller language models, with as few as 2.7B parameters, can
effectively engage in intricate dialogues that integrate both textual and
visual elements, provided they are trained with high-quality corpora. Our model
delivers commendable performance on publicly available benchmarks that
encompass visual comprehension, reasoning, and knowledge-based perception.
Beyond its remarkable performance in multi-modal dialogue tasks, our model
opens new avenues for applications in time-sensitive environments and systems
that require real-time interaction, such as embodied agents. It highlights the
potential of smaller language models to achieve sophisticated levels of
understanding and interaction, while maintaining greater resource
efficiency.The project is available at {https://github.com/zhuyiche/llava-phi}.
A compact multi-modal assistant, LLaVA-Phi, using a smaller language model, Phi-2, achieves high performance in visual and multi-modal dialog tasks with resource efficiency.
multi-modal assistant Phi-2 multi-modal dialogues visual comprehension reasoning knowledge-based perception embodied agents