Language Model Can Listen While Speaking
Dialogue serves as the most natural manner of human-computer interaction
(HCI). Recent advancements in speech language models (SLM) have significantly
enhanced speech-based conversational AI. However, these models are limited to
turn-based conversation, lacking the ability to interact with humans in
real-time spoken scenarios, for example, being interrupted when the generated
content is not satisfactory. To address these limitations, we explore full
duplex modeling (FDM) in interactive speech language models (iSLM), focusing on
enhancing real-time interaction and, more explicitly, exploring the
quintessential ability of interruption. We introduce a novel model design,
namely listening-while-speaking language model (LSLM), an end-to-end system
equipped with both listening and speaking channels. Our LSLM employs a
token-based decoder-only TTS for speech generation and a streaming
self-supervised learning (SSL) encoder for real-time audio input. LSLM fuses
both channels for autoregressive generation and detects turn-taking in real
time. Three fusion strategies -- early fusion, middle fusion, and late fusion
-- are explored, with middle fusion achieving an optimal balance between speech
generation and real-time interaction. Two experimental settings, command-based
FDM and voice-based FDM, demonstrate LSLM's robustness to noise and sensitivity
to diverse instructions. Our results highlight LSLM's capability to achieve
duplex communication with minimal impact on existing systems. This study aims
to advance the development of interactive speech dialogue systems, enhancing
their applicability in real-world contexts.
66b197769aedfcdb8f5c19f9
Ziyang Ma
66b197769aedfcdb8f5c19fa
Yakun Song
66b197769aedfcdb8f5c19fb
Chenpeng Du
admin_assigned
2024-08-06T10:38:48.163Z
661ae5f7040d38d24bdd48e0
/avatars/79a0cc8c5bae3c422160002dbc2869ce.svg
Chenpeng Du
user
cpdu
66b197769aedfcdb8f5c19fc
Jian Cong
66b197769aedfcdb8f5c19fd
Zhuo Chen
66b197769aedfcdb8f5c19fe
Yuping Wang
66b197769aedfcdb8f5c19ff
Yuxuan Wang
admin_assigned
2024-08-06T13:18:23.971Z
6694e5344a34b68738070c07
/avatars/f33ec08ec9822bb353020000a2fcbd84.svg
Yuxuan WANG
user
AlbusYuxuanWang
66b197769aedfcdb8f5c1a00
Xie Chen
Language Model Can Listen While Speaking
Dialogue serves as the most natural manner of human-computer interaction
(HCI). Recent advancements in speech language models (SLM) have significantly
enhanced speech-based conversational AI. However, these models are limited to
turn-based conversation, lacking the ability to interact with humans in
real-time spoken scenarios, for example, being interrupted when the generated
content is not satisfactory. To address these limitations, we explore full
duplex modeling (FDM) in interactive speech language models (iSLM), focusing on
enhancing real-time interaction and, more explicitly, exploring the
quintessential ability of interruption. We introduce a novel model design,
namely listening-while-speaking language model (LSLM), an end-to-end system
equipped with both listening and speaking channels. Our LSLM employs a
token-based decoder-only TTS for speech generation and a streaming
self-supervised learning (SSL) encoder for real-time audio input. LSLM fuses
both channels for autoregressive generation and detects turn-taking in real
time. Three fusion strategies -- early fusion, middle fusion, and late fusion
-- are explored, with middle fusion achieving an optimal balance between speech
generation and real-time interaction. Two experimental settings, command-based
FDM and voice-based FDM, demonstrate LSLM's robustness to noise and sensitivity
to diverse instructions. Our results highlight LSLM's capability to achieve
duplex communication with minimal impact on existing systems. This study aims
to advance the development of interactive speech dialogue systems, enhancing
their applicability in real-world contexts.
A novel listening-while-speaking language model (LSLM) enhances real-time, full-duplex speech interaction by integrating speech generation and real-time audio input with an optimal fusion strategy.
full duplex modeling interactive speech language models listening-while-speaking language model token-based decoder-only TTS streaming self-supervised learning early fusion middle fusion late fusion duplex communication