DiTaiListener: Controllable High Fidelity Listener Video Generation with
  Diffusion
Generating naturalistic and nuanced listener motions for extended
interactions remains an open problem. Existing methods often rely on
low-dimensional motion codes for facial behavior generation followed by
photorealistic rendering, limiting both visual fidelity and expressive
richness. To address these challenges, we introduce DiTaiListener, powered by a
video diffusion model with multimodal conditions. Our approach first generates
short segments of listener responses conditioned on the speaker's speech and
facial motions with DiTaiListener-Gen. It then refines the transitional frames
via DiTaiListener-Edit for a seamless transition. Specifically,
DiTaiListener-Gen adapts a Diffusion Transformer (DiT) for the task of listener
head portrait generation by introducing a Causal Temporal Multimodal Adapter
(CTM-Adapter) to process speakers' auditory and visual cues. CTM-Adapter
integrates speakers' input in a causal manner into the video generation process
to ensure temporally coherent listener responses. For long-form video
generation, we introduce DiTaiListener-Edit, a transition refinement
video-to-video diffusion model. The model fuses video segments into smooth and
continuous videos, ensuring temporal consistency in facial expressions and
image quality when merging short video segments produced by DiTaiListener-Gen.
Quantitatively, DiTaiListener achieves the state-of-the-art performance on
benchmark datasets in both photorealism (+73.8% in FID on RealTalk) and motion
representation (+6.1% in FD metric on VICO) spaces. User studies confirm the
superior performance of DiTaiListener, with the model being the clear
preference in terms of feedback, diversity, and smoothness, outperforming
competitors by a significant margin.
67f766cb1879ad2f13bee3d1
Maksim Siniukov
admin_assigned
2025-04-10T08:10:42.914Z
63c9f93cdfac8071d01ed56f
https://cdn-avatars.huggingface.co/v1/production/uploads/1674180895772-noauth.jpeg
Maksim Siniukov
user
havent-invented
67f766cb1879ad2f13bee3d2
Di Chang
admin_assigned
2025-04-10T08:10:49.652Z
64a5d8219f3b568c202b3137
https://cdn-avatars.huggingface.co/v1/production/uploads/64a5d8219f3b568c202b3137/TI20Z1lWHzZpLsMkayDbU.png
Di Chang
user
Boese0601
67f766cb1879ad2f13bee3d3
Minh Tran
admin_assigned
2025-04-10T08:10:55.537Z
632b6c08ca316c73cd3e4d8c
/avatars/a02aa14823dd729df0267a9b55779edd.svg
Minh Tran
user
minhtran
67f766cb1879ad2f13bee3d4
Hongkun Gong
admin_assigned
2025-04-10T08:11:01.857Z
6605cfc7b85b7b4ea506a33d
/avatars/92abda656abf2298c9d28b9b2e3643a3.svg
Hongkun Gong
user
hongkung
67f766cb1879ad2f13bee3d5
Ashutosh Chaubey
admin_assigned
2025-04-10T08:11:08.391Z
6541185dbd60d2bd193f7999
/avatars/4ce4a0feff9bfbb87e9f40431718ba00.svg
Ashutosh Chaubey
user
chaubeyG
67f766cb1879ad2f13bee3d6
Mohammad Soleymani
admin_assigned
2025-04-10T08:11:15.030Z
65fcb99d383d3f256c3a92d2
/avatars/b85d32f4d7a19816b8d499e05b173ad1.svg
Mohammad Soleymani
user
msoleymani
DiTaiListener: Controllable High Fidelity Listener Video Generation with
  Diffusion
Generating naturalistic and nuanced listener motions for extended
interactions remains an open problem. Existing methods often rely on
low-dimensional motion codes for facial behavior generation followed by
photorealistic rendering, limiting both visual fidelity and expressive
richness. To address these challenges, we introduce DiTaiListener, powered by a
video diffusion model with multimodal conditions. Our approach first generates
short segments of listener responses conditioned on the speaker's speech and
facial motions with DiTaiListener-Gen. It then refines the transitional frames
via DiTaiListener-Edit for a seamless transition. Specifically,
DiTaiListener-Gen adapts a Diffusion Transformer (DiT) for the task of listener
head portrait generation by introducing a Causal Temporal Multimodal Adapter
(CTM-Adapter) to process speakers' auditory and visual cues. CTM-Adapter
integrates speakers' input in a causal manner into the video generation process
to ensure temporally coherent listener responses. For long-form video
generation, we introduce DiTaiListener-Edit, a transition refinement
video-to-video diffusion model. The model fuses video segments into smooth and
continuous videos, ensuring temporal consistency in facial expressions and
image quality when merging short video segments produced by DiTaiListener-Gen.
Quantitatively, DiTaiListener achieves the state-of-the-art performance on
benchmark datasets in both photorealism (+73.8% in FID on RealTalk) and motion
representation (+6.1% in FD metric on VICO) spaces. User studies confirm the
superior performance of DiTaiListener, with the model being the clear
preference in terms of feedback, diversity, and smoothness, outperforming
competitors by a significant margin.
DiTaiListener generates realistic listener motions using a video diffusion model with multimodal conditions, achieving superior photorealism and motion representation compared to existing methods.
video diffusion model multimodal conditions Diffusion Transformer (DiT) Causal Temporal Multimodal Adapter (CTM-Adapter) video-to-video diffusion model short video segments temporal consistency photorealism motion representation RealTalk VICO user studies