Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional
  Talking Portrait Generation
Recent advances in Talking Head Generation (THG) have achieved impressive lip
synchronization and visual quality through diffusion models; yet existing
methods struggle to generate emotionally expressive portraits while preserving
speaker identity. We identify three critical limitations in current emotional
talking head generation: insufficient utilization of audio's inherent emotional
cues, identity leakage in emotion representations, and isolated learning of
emotion correlations. To address these challenges, we propose a novel framework
dubbed as DICE-Talk, following the idea of disentangling identity with emotion,
and then cooperating emotions with similar characteristics. First, we develop a
disentangled emotion embedder that jointly models audio-visual emotional cues
through cross-modal attention, representing emotions as identity-agnostic
Gaussian distributions. Second, we introduce a correlation-enhanced emotion
conditioning module with learnable Emotion Banks that explicitly capture
inter-emotion relationships through vector quantization and attention-based
feature aggregation. Third, we design an emotion discrimination objective that
enforces affective consistency during the diffusion process through
latent-space classification. Extensive experiments on MEAD and HDTF datasets
demonstrate our method's superiority, outperforming state-of-the-art approaches
in emotion accuracy while maintaining competitive lip-sync performance.
Qualitative results and user studies further confirm our method's ability to
generate identity-preserving portraits with rich, correlated emotional
expressions that naturally adapt to unseen identities.
6810746e4be021d4dcd8d4de
Weipeng Tan
6810746e4be021d4dcd8d4df
Chuming Lin
6810746e4be021d4dcd8d4e0
Chengming Xu
claimed_verified
2025-04-30T09:56:46.843Z
652fab9d04a34a9282bf29d6
/avatars/cd5967b37ebb1225e9ae1d46f196e2e2.svg
Chengming Xu
user
ChengmingX
6810746e4be021d4dcd8d4e1
FeiFan Xu
6810746e4be021d4dcd8d4e2
Xiaobin Hu
6810746e4be021d4dcd8d4e3
Xiaozhong Ji
6810746e4be021d4dcd8d4e4
Junwei Zhu
6810746e4be021d4dcd8d4e5
Chengjie Wang
6810746e4be021d4dcd8d4e6
Yanwei Fu
Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional
  Talking Portrait Generation
Recent advances in Talking Head Generation (THG) have achieved impressive lip
synchronization and visual quality through diffusion models; yet existing
methods struggle to generate emotionally expressive portraits while preserving
speaker identity. We identify three critical limitations in current emotional
talking head generation: insufficient utilization of audio's inherent emotional
cues, identity leakage in emotion representations, and isolated learning of
emotion correlations. To address these challenges, we propose a novel framework
dubbed as DICE-Talk, following the idea of disentangling identity with emotion,
and then cooperating emotions with similar characteristics. First, we develop a
disentangled emotion embedder that jointly models audio-visual emotional cues
through cross-modal attention, representing emotions as identity-agnostic
Gaussian distributions. Second, we introduce a correlation-enhanced emotion
conditioning module with learnable Emotion Banks that explicitly capture
inter-emotion relationships through vector quantization and attention-based
feature aggregation. Third, we design an emotion discrimination objective that
enforces affective consistency during the diffusion process through
latent-space classification. Extensive experiments on MEAD and HDTF datasets
demonstrate our method's superiority, outperforming state-of-the-art approaches
in emotion accuracy while maintaining competitive lip-sync performance.
Qualitative results and user studies further confirm our method's ability to
generate identity-preserving portraits with rich, correlated emotional
expressions that naturally adapt to unseen identities.
A novel framework, DICE-Talk, improves emotional expression in talking head generation by disentangling identity and emotion, capturing emotion correlations, and enforcing affective consistency through diffusion models.
diffusion models talking head generation lips synchronization visual quality audio-visual emotional cues cross-modal attention disentangled emotion embedder Gaussian distributions correlation-enhanced emotion conditioning module Emotion Banks vector quantization attention-based feature aggregation emotion discrimination objective latent-space classification