Towards Human-like Multimodal Conversational Agent by Generating
  Engaging Speech
Human conversation involves language, speech, and visual cues, with each
medium providing complementary information. For instance, speech conveys a vibe
or tone not fully captured by text alone. While multimodal LLMs focus on
generating text responses from diverse inputs, less attention has been paid to
generating natural and engaging speech. We propose a human-like agent that
generates speech responses based on conversation mood and responsive style
information. To achieve this, we build a novel MultiSensory Conversation
dataset focused on speech to enable agents to generate natural speech. We then
propose a multimodal LLM-based model for generating text responses and voice
descriptions, which are used to generate speech covering paralinguistic
information. Experimental results demonstrate the effectiveness of utilizing
both visual and audio modalities in conversation to generate engaging speech.
The source code is available in https://github.com/kimtaesu24/MSenC
68ce407d8adc5cd018d1579e
Taesoo Kim
claimed_verified
2025-09-21T13:12:19.904Z
64b4d71965a7e15eac0a825c
/avatars/e77dae65fd58990574fbd47d7bb07b0a.svg
TAESOO KIM
user
TAESOO98
68ce407d8adc5cd018d1579f
Yongsik Jo
68ce407d8adc5cd018d157a0
Hyunmin Song
68ce407d8adc5cd018d157a1
Taehwan Kim
Towards Human-like Multimodal Conversational Agent by Generating
  Engaging Speech
Human conversation involves language, speech, and visual cues, with each
medium providing complementary information. For instance, speech conveys a vibe
or tone not fully captured by text alone. While multimodal LLMs focus on
generating text responses from diverse inputs, less attention has been paid to
generating natural and engaging speech. We propose a human-like agent that
generates speech responses based on conversation mood and responsive style
information. To achieve this, we build a novel MultiSensory Conversation
dataset focused on speech to enable agents to generate natural speech. We then
propose a multimodal LLM-based model for generating text responses and voice
descriptions, which are used to generate speech covering paralinguistic
information. Experimental results demonstrate the effectiveness of utilizing
both visual and audio modalities in conversation to generate engaging speech.
The source code is available in https://github.com/kimtaesu24/MSenC
A multimodal LLM-based model generates natural and engaging speech by integrating visual and audio modalities, using a novel MultiSensory Conversation dataset.
multimodal LLMs MultiSensory Conversation dataset speech generation conversation mood responsive style paralinguistic information