S2S-Arena, Evaluating Speech2Speech Protocols on Instruction Following
  with Paralinguistic Information
The rapid development of large language models (LLMs) has brought significant
attention to speech models, particularly recent progress in speech2speech
protocols supporting speech input and output. However, the existing benchmarks
adopt automatic text-based evaluators for evaluating the instruction following
ability of these models lack consideration for paralinguistic information in
both speech understanding and generation. To address these issues, we introduce
S2S-Arena, a novel arena-style S2S benchmark that evaluates
instruction-following capabilities with paralinguistic information in both
speech-in and speech-out across real-world tasks. We design 154 samples that
fused TTS and live recordings in four domains with 21 tasks and manually
evaluate existing popular speech models in an arena-style manner. The
experimental results show that: (1) in addition to the superior performance of
GPT-4o, the speech model of cascaded ASR, LLM, and TTS outperforms the jointly
trained model after text-speech alignment in speech2speech protocols; (2)
considering paralinguistic information, the knowledgeability of the speech
model mainly depends on the LLM backbone, and the multilingual support of that
is limited by the speech module; (3) excellent speech models can already
understand the paralinguistic information in speech input, but generating
appropriate audio with paralinguistic information is still a challenge.
67ced88fa0db83a841e09be6
Feng Jiang
claimed_verified
2025-03-10T12:46:04.417Z
62d4cdbdb9e2ec814c0e0659
/avatars/616dd89be0fe8937b32c3f23c69a4e15.svg
feng jiang
user
liuxuan320
67ced88fa0db83a841e09be7
Zhiyu Lin
claimed_verified
2025-03-10T13:35:13.930Z
67061847e343e345b777d574
/avatars/6b6f35aa563dcf32937c4b6a3cdd870e.svg
Zhiyu
user
zylin1
67ced88fa0db83a841e09be8
Fan Bu
claimed_verified
2025-03-10T13:35:16.297Z
668e7f46c243a12604035758
/avatars/35bd20032fafb7d7603266cf9a72d1e0.svg
Fan Bu
user
FanBuCUHK
67ced88fa0db83a841e09be9
Yuhao Du
claimed_verified
2025-09-13T15:06:30.315Z
662f495aa55baffc36870052
https://cdn-avatars.huggingface.co/v1/production/uploads/662f495aa55baffc36870052/8335-GrXBOJne4ZNgKrRz.jpeg
Yuhao Du
user
KurtDu
67ced88fa0db83a841e09bea
Benyou Wang
claimed_verified
2025-03-10T13:35:11.668Z
637c6703ca8542a0ba900ccb
/avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
Wang
user
Benyou
67ced88fa0db83a841e09beb
Haizhou Li
S2S-Arena, Evaluating Speech2Speech Protocols on Instruction Following
  with Paralinguistic Information
The rapid development of large language models (LLMs) has brought significant
attention to speech models, particularly recent progress in speech2speech
protocols supporting speech input and output. However, the existing benchmarks
adopt automatic text-based evaluators for evaluating the instruction following
ability of these models lack consideration for paralinguistic information in
both speech understanding and generation. To address these issues, we introduce
S2S-Arena, a novel arena-style S2S benchmark that evaluates
instruction-following capabilities with paralinguistic information in both
speech-in and speech-out across real-world tasks. We design 154 samples that
fused TTS and live recordings in four domains with 21 tasks and manually
evaluate existing popular speech models in an arena-style manner. The
experimental results show that: (1) in addition to the superior performance of
GPT-4o, the speech model of cascaded ASR, LLM, and TTS outperforms the jointly
trained model after text-speech alignment in speech2speech protocols; (2)
considering paralinguistic information, the knowledgeability of the speech
model mainly depends on the LLM backbone, and the multilingual support of that
is limited by the speech module; (3) excellent speech models can already
understand the paralinguistic information in speech input, but generating
appropriate audio with paralinguistic information is still a challenge.
S2S-Arena is introduced to evaluate speech models' instruction-following abilities with paralinguistic information, revealing that cascaded ASR, LLM, and TTS outperform jointly trained models in speech2speech protocols and that generating appropriate audio with paralinguistic information remains challenging.
large language models LLMs speech2speech S2S-Arena paralinguistic information speech understanding speech generation TTS ASR task evaluation speech model knowledgeability multilingual support