VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
Recent Multimodal Large Language Models (MLLMs) have typically focused on
integrating visual and textual modalities, with less emphasis placed on the
role of speech in enhancing interaction. However, speech plays a crucial role
in multimodal dialogue systems, and implementing high-performance in both
vision and speech tasks remains a significant challenge due to the fundamental
modality differences. In this paper, we propose a carefully designed
multi-stage training methodology that progressively trains LLM to understand
both visual and speech information, ultimately enabling fluent vision and
speech interaction. Our approach not only preserves strong vision-language
capacity, but also enables efficient speech-to-speech dialogue capabilities
without separate ASR and TTS modules, significantly accelerating multimodal
end-to-end response speed. By comparing our method against state-of-the-art
counterparts across benchmarks for image, video, and speech tasks, we
demonstrate that our model is equipped with both strong visual and speech
capabilities, making near real-time vision and speech interaction.
677b5979a54b76dcaa4991f9
Chaoyou Fu
677b5979a54b76dcaa4991fa
Haojia Lin
admin_assigned
2025-01-06T08:24:54.539Z
64ffd436d522560505a94b8e
/avatars/02d4faac40ac203cb5d635cfcb39780c.svg
Haojia Lin
user
linhaojia13
677b5979a54b76dcaa4991fb
Xiong Wang
admin_assigned
2025-01-06T08:25:10.743Z
664eaf0a98e93ef417c3cc42
/avatars/67fb44351cac8964410e5b6549817182.svg
Xiong Wang
user
xiongwang
677b5979a54b76dcaa4991fc
Yi-Fan Zhang
admin_assigned
2025-01-06T08:25:17.351Z
623d8ca4c29adf5ef6175615
https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
Yi-Fan Zhang
user
yifanzhang114
677b5979a54b76dcaa4991fd
Yunhang Shen
admin_assigned
2025-01-06T08:25:29.653Z
6483143902f98c3f05aff915
https://cdn-avatars.huggingface.co/v1/production/uploads/6483143902f98c3f05aff915/ZhWFFgrlRsQf4MXiInh5p.jpeg
沈云航 Yunhang Shen
user
shenyunhang
677b5979a54b76dcaa4991fe
Xiaoyu Liu
677b5979a54b76dcaa4991ff
Yangze Li
677b5979a54b76dcaa499200
Zuwei Long
677b5979a54b76dcaa499201
Heting Gao
admin_assigned
2025-01-06T08:26:18.115Z
65ff5dcf82708115869da69a
/avatars/10edebbc559e9fb8b0e377c82eba66d4.svg
Heting Gao
user
hertin
677b5979a54b76dcaa499202
Ke Li
claimed_verified
2025-01-09T10:07:25.742Z
63280915eeee4dd858083092
/avatars/78347af4af42527d53e88d9969c5c934.svg
Ke Li
user
tristanli
677b5979a54b76dcaa499203
Xiawu Zheng
admin_assigned
2025-01-06T08:26:25.079Z
665d85e35491b1e10d0d5221
/avatars/e46ea55b197e2bf8038871ae95f59585.svg
Xiawu Zheng
user
zhengxiawu
677b5979a54b76dcaa499204
Rongrong Ji
677b5979a54b76dcaa499205
Xing Sun
admin_assigned
2025-01-06T08:30:21.180Z
647401e50da364bd0d002f2a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vPuPn7EV092mLBOM2YZXd.png
XING SUN
user
tedsun
677b5979a54b76dcaa499206
Caifeng Shan
677b5979a54b76dcaa499207
Ran He
VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
Recent Multimodal Large Language Models (MLLMs) have typically focused on
integrating visual and textual modalities, with less emphasis placed on the
role of speech in enhancing interaction. However, speech plays a crucial role
in multimodal dialogue systems, and implementing high-performance in both
vision and speech tasks remains a significant challenge due to the fundamental
modality differences. In this paper, we propose a carefully designed
multi-stage training methodology that progressively trains LLM to understand
both visual and speech information, ultimately enabling fluent vision and
speech interaction. Our approach not only preserves strong vision-language
capacity, but also enables efficient speech-to-speech dialogue capabilities
without separate ASR and TTS modules, significantly accelerating multimodal
end-to-end response speed. By comparing our method against state-of-the-art
counterparts across benchmarks for image, video, and speech tasks, we
demonstrate that our model is equipped with both strong visual and speech
capabilities, making near real-time vision and speech interaction.
A multi-stage training method enhances multimodal large language models with both visual and speech capabilities, enabling real-time interaction without separate ASR and TTS modules.
Multimodal Large Language Models multi-stage training speech-to-speech dialogue vision-language capacity ASR TTS