LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive
  Streaming Speech Synthesis
Real-time, intelligent, and natural speech interaction is an essential part
of the next-generation human-computer interaction. Recent advancements have
showcased the potential of building intelligent spoken chatbots based on large
language models (LLMs). In this paper, we introduce LLaMA-Omni 2, a series of
speech language models (SpeechLMs) ranging from 0.5B to 14B parameters, capable
of achieving high-quality real-time speech interaction. LLaMA-Omni 2 is built
upon the Qwen2.5 series models, integrating a speech encoder and an
autoregressive streaming speech decoder. Despite being trained on only 200K
multi-turn speech dialogue samples, LLaMA-Omni 2 demonstrates strong
performance on several spoken question answering and speech instruction
following benchmarks, surpassing previous state-of-the-art SpeechLMs like
GLM-4-Voice, which was trained on millions of hours of speech data.
681975abba26bf20601bb7ca
Qingkai Fang
admin_assigned
2025-05-06T09:15:43.683Z
65b7573482d384513443875e
/avatars/0f2175e4adf507f5ccb0636c1cb647de.svg
Qingkai Fang
user
poeroz
681975abba26bf20601bb7cb
Yan Zhou
681975abba26bf20601bb7cc
Shoutao Guo
admin_assigned
2025-05-06T09:15:53.154Z
66680c0505c407bfea87667c
/avatars/e3c26d2eb13fe8ad2b3fd16897e61e6d.svg
Shoutao Guo
user
guoshoutao
681975abba26bf20601bb7cd
Shaolei Zhang
admin_assigned
2025-05-06T09:16:00.139Z
64803e5dc57f629056c601f1
/avatars/a9e9c97c70714e3a29bef2cf929ee6b3.svg
Shaolei Zhang
user
zhangshaolei
681975abba26bf20601bb7ce
Yang Feng
LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive
  Streaming Speech Synthesis
Real-time, intelligent, and natural speech interaction is an essential part
of the next-generation human-computer interaction. Recent advancements have
showcased the potential of building intelligent spoken chatbots based on large
language models (LLMs). In this paper, we introduce LLaMA-Omni 2, a series of
speech language models (SpeechLMs) ranging from 0.5B to 14B parameters, capable
of achieving high-quality real-time speech interaction. LLaMA-Omni 2 is built
upon the Qwen2.5 series models, integrating a speech encoder and an
autoregressive streaming speech decoder. Despite being trained on only 200K
multi-turn speech dialogue samples, LLaMA-Omni 2 demonstrates strong
performance on several spoken question answering and speech instruction
following benchmarks, surpassing previous state-of-the-art SpeechLMs like
GLM-4-Voice, which was trained on millions of hours of speech data.
LLaMA-Omni 2, a series of speech language models with parameters ranging from 0.5B to 14B, achieves high-quality real-time speech interaction through a speech encoder and autoregressive streaming speech decoder, outperforming models like GLM-4-Voice with significantly less training data.
speech language models SpeechLMs Qwen2.5 speech encoder autoregressive streaming speech decoder spoken question answering speech instruction following GLM-4-Voice