VoiceAssistant-Eval: Benchmarking AI Assistants across Listening,
  Speaking, and Viewing
The growing capabilities of large language models and multimodal systems have
spurred interest in voice-first AI assistants, yet existing benchmarks are
inadequate for evaluating the full range of these systems' capabilities. We
introduce VoiceAssistant-Eval, a comprehensive benchmark designed to assess AI
assistants across listening, speaking, and viewing. VoiceAssistant-Eval
comprises 10,497 curated examples spanning 13 task categories. These tasks
include natural sounds, music, and spoken dialogue for listening; multi-turn
dialogue, role-play imitation, and various scenarios for speaking; and highly
heterogeneous images for viewing. To demonstrate its utility, we evaluate 21
open-source models and GPT-4o-Audio, measuring the quality of the response
content and speech, as well as their consistency. The results reveal three key
findings: (1) proprietary models do not universally outperform open-source
models; (2) most models excel at speaking tasks but lag in audio understanding;
and (3) well-designed smaller models can rival much larger ones. Notably, the
mid-sized Step-Audio-2-mini (7B) achieves more than double the listening
accuracy of LLaMA-Omni2-32B-Bilingual. However, challenges remain: multimodal
(audio plus visual) input and role-play voice imitation tasks are difficult for
current models, and significant gaps persist in robustness and safety
alignment. VoiceAssistant-Eval identifies these gaps and establishes a rigorous
framework for evaluating and guiding the development of next-generation AI
assistants. Code and data will be released at
https://mathllm.github.io/VoiceAssistantEval/ .
68d9ebbd0177a6054b013a51
Ke Wang
claimed_verified
2025-09-29T05:58:58.098Z
64d592c28767727dffa1f002
/avatars/fe38bcac944a2742dc12c624e62d24ef.svg
WangKe
user
scikkk
68d9ebbd0177a6054b013a52
Houxing Ren
68d9ebbd0177a6054b013a53
Zimu Lu
68d9ebbd0177a6054b013a54
Mingjie Zhan
68d9ebbd0177a6054b013a55
Hongsheng Li
VoiceAssistant-Eval: Benchmarking AI Assistants across Listening,
  Speaking, and Viewing
The growing capabilities of large language models and multimodal systems have
spurred interest in voice-first AI assistants, yet existing benchmarks are
inadequate for evaluating the full range of these systems' capabilities. We
introduce VoiceAssistant-Eval, a comprehensive benchmark designed to assess AI
assistants across listening, speaking, and viewing. VoiceAssistant-Eval
comprises 10,497 curated examples spanning 13 task categories. These tasks
include natural sounds, music, and spoken dialogue for listening; multi-turn
dialogue, role-play imitation, and various scenarios for speaking; and highly
heterogeneous images for viewing. To demonstrate its utility, we evaluate 21
open-source models and GPT-4o-Audio, measuring the quality of the response
content and speech, as well as their consistency. The results reveal three key
findings: (1) proprietary models do not universally outperform open-source
models; (2) most models excel at speaking tasks but lag in audio understanding;
and (3) well-designed smaller models can rival much larger ones. Notably, the
mid-sized Step-Audio-2-mini (7B) achieves more than double the listening
accuracy of LLaMA-Omni2-32B-Bilingual. However, challenges remain: multimodal
(audio plus visual) input and role-play voice imitation tasks are difficult for
current models, and significant gaps persist in robustness and safety
alignment. VoiceAssistant-Eval identifies these gaps and establishes a rigorous
framework for evaluating and guiding the development of next-generation AI
assistants. Code and data will be released at
https://mathllm.github.io/VoiceAssistantEval/ .
VoiceAssistant-Eval is a benchmark for evaluating AI assistants across listening, speaking, and viewing tasks, revealing insights into model performance and identifying areas for improvement.
large language models multimodal systems voice-first AI assistants VoiceAssistant-Eval listening speaking viewing natural sounds music spoken dialogue multi-turn dialogue role-play imitation highly heterogeneous images response content speech consistency proprietary models open-source models GPT-4o-Audio listening accuracy Step-Audio-2-mini LLaMA-Omni2-32B-Bilingual multimodal input role-play voice imitation robustness safety alignment