OmniResponse: Online Multimodal Conversational Response Generation in
  Dyadic Interactions
In this paper, we introduce Online Multimodal Conversational Response
Generation (OMCRG), a novel task that aims to online generate synchronized
verbal and non-verbal listener feedback, conditioned on the speaker's
multimodal input. OMCRG reflects natural dyadic interactions and poses new
challenges in achieving synchronization between the generated audio and facial
responses of the listener. To address these challenges, we innovatively
introduce text as an intermediate modality to bridge the audio and facial
responses. We hence propose OmniResponse, a Multimodal Large Language Model
(MLLM) that autoregressively generates high-quality multi-modal listener
responses. OmniResponse leverages a pretrained LLM enhanced with two novel
components: Chrono-Text, which temporally anchors generated text tokens, and
TempoVoice, a controllable online TTS module that produces speech synchronized
with facial reactions. To support further OMCRG research, we present
ResponseNet, a new dataset comprising 696 high-quality dyadic interactions
featuring synchronized split-screen videos, multichannel audio, transcripts,
and facial behavior annotations. Comprehensive evaluations conducted on
ResponseNet demonstrate that OmniResponse significantly outperforms baseline
models in terms of semantic speech content, audio-visual synchronization, and
generation quality.
683b44583f2842f6afcc5e6f
Cheng Luo
683b44583f2842f6afcc5e70
Jianghui Wang
683b44583f2842f6afcc5e71
Bing Li
admin_assigned
2025-06-03T12:45:31.879Z
666ddb45c0f3d5afc27e85ba
/avatars/dce98fc77bd8cf9e348f2d91bc3c0225.svg
Bing Li
user
bing-li-ai
683b44583f2842f6afcc5e72
Siyang Song
683b44583f2842f6afcc5e73
Bernard Ghanem
admin_assigned
2025-06-03T12:45:46.322Z
6808bf97ffadd78ec71cb721
/avatars/9adca3142c06b8f69889fcbe85fa374d.svg
Bernard Ghanem
user
bernardghanem
OmniResponse: Online Multimodal Conversational Response Generation in
  Dyadic Interactions
In this paper, we introduce Online Multimodal Conversational Response
Generation (OMCRG), a novel task that aims to online generate synchronized
verbal and non-verbal listener feedback, conditioned on the speaker's
multimodal input. OMCRG reflects natural dyadic interactions and poses new
challenges in achieving synchronization between the generated audio and facial
responses of the listener. To address these challenges, we innovatively
introduce text as an intermediate modality to bridge the audio and facial
responses. We hence propose OmniResponse, a Multimodal Large Language Model
(MLLM) that autoregressively generates high-quality multi-modal listener
responses. OmniResponse leverages a pretrained LLM enhanced with two novel
components: Chrono-Text, which temporally anchors generated text tokens, and
TempoVoice, a controllable online TTS module that produces speech synchronized
with facial reactions. To support further OMCRG research, we present
ResponseNet, a new dataset comprising 696 high-quality dyadic interactions
featuring synchronized split-screen videos, multichannel audio, transcripts,
and facial behavior annotations. Comprehensive evaluations conducted on
ResponseNet demonstrate that OmniResponse significantly outperforms baseline
models in terms of semantic speech content, audio-visual synchronization, and
generation quality.
OmniResponse, a Multimodal Large Language Model, generates high-quality synchronized verbal and non-verbal listener responses using text as an intermediate modality.
Online Multimodal Conversational Response Generation OmniResponse Multimodal Large Language Model Chrono-Text TempoVoice ResponseNet audio-visual synchronization