On decoder-only architecture for speech-to-text and large language model
  integration
Large language models (LLMs) have achieved remarkable success in the field of
natural language processing, enabling better human-computer interaction using
natural language. However, the seamless integration of speech signals into LLMs
has not been explored well. The "decoder-only" architecture has also not been
well studied for speech processing tasks. In this research, we introduce
Speech-LLaMA, a novel approach that effectively incorporates acoustic
information into text-based large language models. Our method leverages
Connectionist Temporal Classification and a simple audio encoder to map the
compressed acoustic features to the continuous semantic space of the LLM. In
addition, we further probe the decoder-only architecture for speech-to-text
tasks by training a smaller scale randomly initialized speech-LLaMA model from
speech-text paired data alone. We conduct experiments on multilingual
speech-to-text translation tasks and demonstrate a significant improvement over
strong baselines, highlighting the potential advantages of decoder-only models
for speech-to-text conversion.
64acb4b667c1cee8dfde2349
Jian Wu
64acb4b667c1cee8dfde234a
Yashesh Gaur
64acb4b667c1cee8dfde234b
Zhuo Chen
claimed_verified
2023-07-12T07:29:41.641Z
642635d22a423daad52bf3e2
/avatars/dbcb757428a1b9d275c116dd56852f98.svg
Zhuo Chen
user
pqpq123
64acb4b667c1cee8dfde234c
Long Zhou
64acb4b667c1cee8dfde234d
Yimeng Zhu
64acb4b667c1cee8dfde234e
Tianrui Wang
64acb4b667c1cee8dfde234f
Jinyu Li
64acb4b667c1cee8dfde2350
Shujie Liu
64acb4b667c1cee8dfde2351
Bo Ren
claimed_verified
2025-03-04T08:54:12.648Z
668dcf92835bf7e64bbca904
/avatars/416eb3a3c5318a6a45aad87012296470.svg
Bo Ren
user
rosrad
64acb4b667c1cee8dfde2352
Linquan Liu
64acb4b667c1cee8dfde2353
Yu Wu
On decoder-only architecture for speech-to-text and large language model
  integration
Large language models (LLMs) have achieved remarkable success in the field of
natural language processing, enabling better human-computer interaction using
natural language. However, the seamless integration of speech signals into LLMs
has not been explored well. The "decoder-only" architecture has also not been
well studied for speech processing tasks. In this research, we introduce
Speech-LLaMA, a novel approach that effectively incorporates acoustic
information into text-based large language models. Our method leverages
Connectionist Temporal Classification and a simple audio encoder to map the
compressed acoustic features to the continuous semantic space of the LLM. In
addition, we further probe the decoder-only architecture for speech-to-text
tasks by training a smaller scale randomly initialized speech-LLaMA model from
speech-text paired data alone. We conduct experiments on multilingual
speech-to-text translation tasks and demonstrate a significant improvement over
strong baselines, highlighting the potential advantages of decoder-only models
for speech-to-text conversion.
Speech-LLaMA integrates acoustic information into text-based LLMs using Connectionist Temporal Classification and a simple audio encoder, demonstrating improved performance on multilingual speech-to-text tasks using a decoder-only architecture.
Speech-LLaMA Connectionist Temporal Classification audio encoder semantic space decoder-only architecture speech-to-text multilingual speech-to-text translation