EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for
  Speech-to-Speech LLMs
Speech-to-speech large language models (SLLMs) are attracting increasing
attention. Derived from text-based large language models (LLMs), SLLMs often
exhibit degradation in knowledge and reasoning capabilities. We hypothesize
that this limitation arises because current training paradigms for SLLMs fail
to bridge the acoustic-semantic gap in the feature representation space. To
address this issue, we propose EchoX, which leverages semantic representations
and dynamically generates speech training targets. This approach integrates
both acoustic and semantic learning, enabling EchoX to preserve strong
reasoning abilities as a speech LLM. Experimental results demonstrate that
EchoX, with about six thousand hours of training data, achieves advanced
performance on multiple knowledge-based question-answering benchmarks. The
project is available at https://github.com/FreedomIntelligence/EchoX.
68c38053fc1747b9124039ea
Yuhao Zhang
claimed_verified
2025-09-12T16:08:19.144Z
66975b9f8031bf92b428e138
/avatars/3254281a7bac1c8ddde1d6bc7e518b2f.svg
Yuhao Zhang
user
Yoohao
68c38053fc1747b9124039eb
Yuhao Du
claimed_verified
2025-09-13T14:56:20.609Z
662f495aa55baffc36870052
https://cdn-avatars.huggingface.co/v1/production/uploads/662f495aa55baffc36870052/8335-GrXBOJne4ZNgKrRz.jpeg
Yuhao Du
user
KurtDu
68c38053fc1747b9124039ec
Zhanchen Dai
claimed_verified
2025-09-15T15:08:41.197Z
671ba4e9b70b3cf8014b728c
/avatars/6ab84d04aa8ca17f4f83c9b93548e89d.svg
Zhanchen Dai
user
tzzte
68c38053fc1747b9124039ed
Xiangnan Ma
68c38053fc1747b9124039ee
Kaiqi Kou
68c38053fc1747b9124039ef
Benyou Wang
68c38053fc1747b9124039f0
Haizhou Li
EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for
  Speech-to-Speech LLMs
Speech-to-speech large language models (SLLMs) are attracting increasing
attention. Derived from text-based large language models (LLMs), SLLMs often
exhibit degradation in knowledge and reasoning capabilities. We hypothesize
that this limitation arises because current training paradigms for SLLMs fail
to bridge the acoustic-semantic gap in the feature representation space. To
address this issue, we propose EchoX, which leverages semantic representations
and dynamically generates speech training targets. This approach integrates
both acoustic and semantic learning, enabling EchoX to preserve strong
reasoning abilities as a speech LLM. Experimental results demonstrate that
EchoX, with about six thousand hours of training data, achieves advanced
performance on multiple knowledge-based question-answering benchmarks. The
project is available at https://github.com/FreedomIntelligence/EchoX.
EchoX, a speech-to-speech large language model, addresses the acoustic-semantic gap by integrating semantic representations, preserving reasoning abilities, and achieving advanced performance on knowledge-based benchmarks.
speech-to-speech large language models SLLMs text-based large language models LLMs acoustic-semantic gap semantic representations speech training targets acoustic learning semantic learning knowledge-based question-answering benchmarks