VITA: Towards Open-Source Interactive Omni Multimodal LLM
The remarkable multimodal capabilities and interactive experience of GPT-4o
underscore their necessity in practical applications, yet open-source models
rarely excel in both areas. In this paper, we introduce VITA, the first-ever
open-source Multimodal Large Language Model (MLLM) adept at simultaneous
processing and analysis of Video, Image, Text, and Audio modalities, and
meanwhile has an advanced multimodal interactive experience. Starting from
Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary
followed by bilingual instruction tuning. We further endow the language model
with visual and audio capabilities through two-stage multi-task learning of
multimodal alignment and instruction tuning. VITA demonstrates robust
foundational capabilities of multilingual, vision, and audio understanding, as
evidenced by its strong performance across a range of both unimodal and
multimodal benchmarks. Beyond foundational capabilities, we have made
considerable progress in enhancing the natural multimodal human-computer
interaction experience. To the best of our knowledge, we are the first to
exploit non-awakening interaction and audio interrupt in MLLM. VITA is the
first step for the open-source community to explore the seamless integration of
multimodal understanding and interaction. While there is still lots of work to
be done on VITA to get close to close-source counterparts, we hope that its
role as a pioneer can serve as a cornerstone for subsequent research. Project
Page: https://vita-home.github.io.
66b982ca115bf3ff568c5dd1
Chaoyou Fu
66b982ca115bf3ff568c5dd2
Haojia Lin
admin_assigned
2024-08-13T09:54:42.395Z
64ffd436d522560505a94b8e
/avatars/02d4faac40ac203cb5d635cfcb39780c.svg
Haojia Lin
user
linhaojia13
66b982ca115bf3ff568c5dd3
Zuwei Long
66b982ca115bf3ff568c5dd4
Yunhang Shen
admin_assigned
2024-08-13T09:55:05.630Z
6483143902f98c3f05aff915
https://cdn-avatars.huggingface.co/v1/production/uploads/6483143902f98c3f05aff915/ZhWFFgrlRsQf4MXiInh5p.jpeg
沈云航 Yunhang Shen
user
shenyunhang
66b982ca115bf3ff568c5dd5
Meng Zhao
claimed_verified
2024-08-13T07:38:30.043Z
64b77da86d1ab56458452ee0
/avatars/0523ec058e4cdc94af543af5dece9c06.svg
Meng Zhao
user
Vector8421
66b982ca115bf3ff568c5dd6
Yifan Zhang
admin_assigned
2024-08-13T09:55:44.449Z
623d8ca4c29adf5ef6175615
https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
Yi-Fan Zhang
user
yifanzhang114
66b982ca115bf3ff568c5dd7
Xiong Wang
66b982ca115bf3ff568c5dd8
Di Yin
66b982ca115bf3ff568c5dd9
Long Ma
66b982ca115bf3ff568c5dda
Xiawu Zheng
admin_assigned
2024-08-13T09:56:47.955Z
665d85e35491b1e10d0d5221
/avatars/e46ea55b197e2bf8038871ae95f59585.svg
Xiawu Zheng
user
zhengxiawu
66b982ca115bf3ff568c5ddb
Ran He
66b982ca115bf3ff568c5ddc
Rongrong Ji
66b982ca115bf3ff568c5ddd
Yunsheng Wu
66b982ca115bf3ff568c5dde
Caifeng Shan
66b982ca115bf3ff568c5ddf
Xing Sun
admin_assigned
2024-08-13T09:57:44.428Z
647401e50da364bd0d002f2a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vPuPn7EV092mLBOM2YZXd.png
XING SUN
user
tedsun
VITA: Towards Open-Source Interactive Omni Multimodal LLM
The remarkable multimodal capabilities and interactive experience of GPT-4o
underscore their necessity in practical applications, yet open-source models
rarely excel in both areas. In this paper, we introduce VITA, the first-ever
open-source Multimodal Large Language Model (MLLM) adept at simultaneous
processing and analysis of Video, Image, Text, and Audio modalities, and
meanwhile has an advanced multimodal interactive experience. Starting from
Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary
followed by bilingual instruction tuning. We further endow the language model
with visual and audio capabilities through two-stage multi-task learning of
multimodal alignment and instruction tuning. VITA demonstrates robust
foundational capabilities of multilingual, vision, and audio understanding, as
evidenced by its strong performance across a range of both unimodal and
multimodal benchmarks. Beyond foundational capabilities, we have made
considerable progress in enhancing the natural multimodal human-computer
interaction experience. To the best of our knowledge, we are the first to
exploit non-awakening interaction and audio interrupt in MLLM. VITA is the
first step for the open-source community to explore the seamless integration of
multimodal understanding and interaction. While there is still lots of work to
be done on VITA to get close to close-source counterparts, we hope that its
role as a pioneer can serve as a cornerstone for subsequent research. Project
Page: https://vita-home.github.io.
VITA, an open-source Multimodal Large Language Model, excels in processing Video, Image, Text, and Audio with seamless interaction, showcasing advancements in multimodal understanding and human-computer interaction.
Multimodal Large Language Model Mixtral bilingual instruction tuning multi-task learning multimodal alignment multilingual understanding vision understanding audio understanding non-awakening interaction audio interrupt multimodal benchmarks multimodal human-computer interaction