EMOVA: Empowering Language Models to See, Hear and Speak with Vivid
  Emotions
GPT-4o, an omni-modal model that enables vocal conversations with diverse
emotions and tones, marks a milestone for omni-modal foundation models.
However, empowering Large Language Models to perceive and generate images,
texts, and speeches end-to-end with publicly available data remains challenging
in the open-source community. Existing vision-language models rely on external
tools for the speech processing, while speech-language models still suffer from
limited or even without vision-understanding abilities. To address this gap, we
propose EMOVA (EMotionally Omni-present Voice Assistant), to enable Large
Language Models with end-to-end speech capabilities while maintaining the
leading vision-language performance. With a semantic-acoustic disentangled
speech tokenizer, we notice surprisingly that omni-modal alignment can further
enhance vision-language and speech abilities compared with the corresponding
bi-modal aligned counterparts. Moreover, a lightweight style module is proposed
for flexible speech style controls (e.g., emotions and pitches). For the first
time, EMOVA achieves state-of-the-art performance on both the vision-language
and speech benchmarks, and meanwhile, supporting omni-modal spoken dialogue
with vivid emotions.
66f61f0b5e1ef72590bbc638
Kai Chen
claimed_verified
2024-09-27T06:58:09.820Z
65bf3b304b5f8c270d50d6c8
https://cdn-avatars.huggingface.co/v1/production/uploads/65bf3b304b5f8c270d50d6c8/iSC_Em-JtfcsKFoGw4KFc.jpeg
Kai Chen
user
KaiChen1998
66f61f0b5e1ef72590bbc639
Yunhao Gou
admin_assigned
2024-09-27T09:32:01.795Z
6181435169071e2bddc65578
/avatars/c898f76c13d380f47b131bcb6289bb9a.svg
Yunhao Gou
user
gyhdog
66f61f0b5e1ef72590bbc63a
Runhui Huang
claimed_verified
2024-09-30T08:07:12.129Z
630f0542cc8ed75decb03b68
/avatars/f76c3603b2700591f33a8a931f7ca664.svg
huangrh9
user
huangrh9
66f61f0b5e1ef72590bbc63b
Zhili Liu
admin_assigned
2024-09-27T09:33:02.396Z
64f395cef2eeda919ab980a5
/avatars/f92116eaf94defd9873d9cdb4f98c598.svg
ZHILI LIU
user
zhili-liu
66f61f0b5e1ef72590bbc63c
Daxin Tan
claimed_verified
2024-09-30T08:07:09.619Z
6458b6508aa54fb020f5c66c
/avatars/66c8715fece349b579652989a2c79e41.svg
Tan
user
Darcy0123
66f61f0b5e1ef72590bbc63d
Jing Xu
66f61f0b5e1ef72590bbc63e
Chunwei Wang
admin_assigned
2024-09-27T09:33:32.389Z
644131e5666947ad561349c1
/avatars/8a2d9be81cef30cb3bc71a695c802d2f.svg
Wangchunwei
user
17day
66f61f0b5e1ef72590bbc63f
Yi Zhu
66f61f0b5e1ef72590bbc640
Yihan Zeng
admin_assigned
2024-09-27T09:33:41.204Z
65beb318358734fd090f742e
/avatars/962420a21b2aa8689dd0e1d4531fbf35.svg
Yihan Zeng
user
vikyzeng2
66f61f0b5e1ef72590bbc641
Kuo Yang
admin_assigned
2024-09-27T09:33:47.438Z
6366c38a95204b4649cc582a
/avatars/ab97094994b6e82f047b43e436b9e852.svg
Kuo Yang
user
kuoyang1999
66f61f0b5e1ef72590bbc642
Dingdong Wang
66f61f0b5e1ef72590bbc643
Kun Xiang
66f61f0b5e1ef72590bbc644
Haoyuan Li
66f61f0b5e1ef72590bbc645
Haoli Bai
66f61f0b5e1ef72590bbc646
Jianhua Han
66f61f0b5e1ef72590bbc647
Xiaohui Li
66f61f0b5e1ef72590bbc648
Weike Jin
66f61f0b5e1ef72590bbc649
Nian Xie
66f61f0b5e1ef72590bbc64a
Yu Zhang
66f61f0b5e1ef72590bbc64b
James T. Kwok
66f61f0b5e1ef72590bbc64c
Hengshuang Zhao
66f61f0b5e1ef72590bbc64d
Xiaodan Liang
66f61f0b5e1ef72590bbc64e
Dit-Yan Yeung
66f61f0b5e1ef72590bbc64f
Xiao Chen
66f61f0b5e1ef72590bbc650
Zhenguo Li
66f61f0b5e1ef72590bbc651
Wei Zhang
66f61f0b5e1ef72590bbc652
Qun Liu
66f61f0b5e1ef72590bbc653
Lanqing Hong
66f61f0b5e1ef72590bbc654
Lu Hou
66f61f0b5e1ef72590bbc655
Hang Xu
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid
  Emotions
GPT-4o, an omni-modal model that enables vocal conversations with diverse
emotions and tones, marks a milestone for omni-modal foundation models.
However, empowering Large Language Models to perceive and generate images,
texts, and speeches end-to-end with publicly available data remains challenging
in the open-source community. Existing vision-language models rely on external
tools for the speech processing, while speech-language models still suffer from
limited or even without vision-understanding abilities. To address this gap, we
propose EMOVA (EMotionally Omni-present Voice Assistant), to enable Large
Language Models with end-to-end speech capabilities while maintaining the
leading vision-language performance. With a semantic-acoustic disentangled
speech tokenizer, we notice surprisingly that omni-modal alignment can further
enhance vision-language and speech abilities compared with the corresponding
bi-modal aligned counterparts. Moreover, a lightweight style module is proposed
for flexible speech style controls (e.g., emotions and pitches). For the first
time, EMOVA achieves state-of-the-art performance on both the vision-language
and speech benchmarks, and meanwhile, supporting omni-modal spoken dialogue
with vivid emotions.
EMOVA integrates speech, vision, and language capabilities, achieving top performance in benchmarks and enabling emotive spoken dialogue using a disentangled speech tokenizer and style module.
omni-modal model Large Language Models end-to-end speech capabilities view-language models semantic-acoustic disentangled speech tokenizer lightweight style module vision-language benchmarks speech benchmarks omni-modal spoken dialogue