Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex
  Models
As large language models (LLMs) increasingly permeate daily lives, there is a
growing demand for real-time interactions that mirror human conversations.
Traditional turn-based chat systems driven by LLMs prevent users from verbally
interacting with the system while it is generating responses. To overcome these
limitations, we adapt existing LLMs to duplex models so that these
LLMs can listen for users while generating output and dynamically adjust
themselves to provide users with instant feedback. % such as in response to
interruptions. Specifically, we divide the queries and responses of
conversations into several time slices and then adopt a
time-division-multiplexing (TDM) encoding-decoding strategy to
pseudo-simultaneously process these slices. Furthermore, to make LLMs
proficient enough to handle real-time conversations, we build a fine-tuning
dataset consisting of alternating time slices of queries and responses as well
as covering typical feedback types in instantaneous interactions. Our
experiments show that although the queries and responses of conversations are
segmented into incomplete slices for processing, LLMs can preserve their
original performance on standard benchmarks with a few fine-tuning steps on our
dataset. Automatic and human evaluation indicate that duplex models make
user-AI interactions more natural and human-like, and greatly improve user
satisfaction compared to vanilla LLMs. Our duplex model and dataset will be
released.
667a90b03bf5064fe3367191
Xinrong Zhang
admin_assigned
2024-06-25T13:59:38.360Z
6316eec27b0ee0136e5e8132
/avatars/a08f7cbaf155d8cef3dd6197a21890a5.svg
xinrongzhang
user
xinrongzhang2022
667a90b03bf5064fe3367192
Yingfa Chen
admin_assigned
2024-06-25T13:59:44.450Z
6144e4667f2544bb450787b2
https://cdn-avatars.huggingface.co/v1/production/uploads/6144e4667f2544bb450787b2/5wSDDqJbI4TGtBLP9IvjY.png
Yingfa Chen
user
chen-yingfa
667a90b03bf5064fe3367193
Shengding Hu
admin_assigned
2024-06-25T13:59:50.264Z
6041bec6c15e823a685f2afc
/avatars/7d09fbe771d3c3312ac7eb6ef83ce053.svg
Shengding Hu
user
ShengdingHu
667a90b03bf5064fe3367194
Xu Han
667a90b03bf5064fe3367195
Zihang Xu
667a90b03bf5064fe3367196
Yuanwei Xu
667a90b03bf5064fe3367197
Weilin Zhao
admin_assigned
2024-06-25T14:00:34.382Z
64b8ff3d95bd42c770878042
/avatars/564a4dccdf9e5b813a99979b0ef58183.svg
Weilin Zhao
user
Achazwl
667a90b03bf5064fe3367198
Maosong Sun
667a90b03bf5064fe3367199
Zhiyuan Liu
admin_assigned
2024-06-25T14:01:19.677Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex
  Models
As large language models (LLMs) increasingly permeate daily lives, there is a
growing demand for real-time interactions that mirror human conversations.
Traditional turn-based chat systems driven by LLMs prevent users from verbally
interacting with the system while it is generating responses. To overcome these
limitations, we adapt existing LLMs to duplex models so that these
LLMs can listen for users while generating output and dynamically adjust
themselves to provide users with instant feedback. % such as in response to
interruptions. Specifically, we divide the queries and responses of
conversations into several time slices and then adopt a
time-division-multiplexing (TDM) encoding-decoding strategy to
pseudo-simultaneously process these slices. Furthermore, to make LLMs
proficient enough to handle real-time conversations, we build a fine-tuning
dataset consisting of alternating time slices of queries and responses as well
as covering typical feedback types in instantaneous interactions. Our
experiments show that although the queries and responses of conversations are
segmented into incomplete slices for processing, LLMs can preserve their
original performance on standard benchmarks with a few fine-tuning steps on our
dataset. Automatic and human evaluation indicate that duplex models make
user-AI interactions more natural and human-like, and greatly improve user
satisfaction compared to vanilla LLMs. Our duplex model and dataset will be
released.
Duplex models enable real-time, simultaneous interactions for LLMs through time-division-multiplexing, improving user satisfaction and naturalness in AI conversations.
LLMs duplex models time-division-multiplexing (TDM) encoding-decoding strategy time slices fine-tuning standard benchmarks user-AI interactions