Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional
  Talking Portrait Generation
Recent advances in Talking Head Generation (THG) have achieved impressive lip
synchronization and visual quality through diffusion models; yet existing
methods struggle to generate emotionally expressive portraits while preserving
speaker identity. We identify three critical limitations in current emotional
talking head generation: insufficient utilization of audio's inherent emotional
cues, identity leakage in emotion representations, and isolated learning of
emotion correlations. To address these challenges, we propose a novel framework
dubbed as DICE-Talk, following the idea of disentangling identity with emotion,
and then cooperating emotions with similar characteristics. First, we develop a
disentangled emotion embedder that jointly models audio-visual emotional cues
through cross-modal attention, representing emotions as identity-agnostic
Gaussian distributions. Second, we introduce a correlation-enhanced emotion
conditioning module with learnable Emotion Banks that explicitly capture
inter-emotion relationships through vector quantization and attention-based
feature aggregation. Third, we design an emotion discrimination objective that
enforces affective consistency during the diffusion process through
latent-space classification. Extensive experiments on MEAD and HDTF datasets
demonstrate our method's superiority, outperforming state-of-the-art approaches
in emotion accuracy while maintaining competitive lip-sync performance.
Qualitative results and user studies further confirm our method's ability to
generate identity-preserving portraits with rich, correlated emotional
expressions that naturally adapt to unseen identities.
6810746e4be021d4dcd8d4de
Weipeng Tan
6810746e4be021d4dcd8d4df
Chuming Lin
6810746e4be021d4dcd8d4e0
Chengming Xu
claimed_verified
2025-04-30T09:56:46.843Z
652fab9d04a34a9282bf29d6
/avatars/cd5967b37ebb1225e9ae1d46f196e2e2.svg
Chengming Xu
user
ChengmingX
6810746e4be021d4dcd8d4e1
FeiFan Xu
6810746e4be021d4dcd8d4e2
Xiaobin Hu
6810746e4be021d4dcd8d4e3
Xiaozhong Ji
6810746e4be021d4dcd8d4e4
Junwei Zhu
6810746e4be021d4dcd8d4e5
Chengjie Wang
6810746e4be021d4dcd8d4e6
Yanwei Fu
Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional
  Talking Portrait Generation
Recent advances in Talking Head Generation (THG) have achieved impressive lip
synchronization and visual quality through diffusion models; yet existing
methods struggle to generate emotionally expressive portraits while preserving
speaker identity. We identify three critical limitations in current emotional
talking head generation: insufficient utilization of audio's inherent emotional
cues, identity leakage in emotion representations, and isolated learning of
emotion correlations. To address these challenges, we propose a novel framework
dubbed as DICE-Talk, following the idea of disentangling identity with emotion,
and then cooperating emotions with similar characteristics. First, we develop a
disentangled emotion embedder that jointly models audio-visual emotional cues
through cross-modal attention, representing emotions as identity-agnostic
Gaussian distributions. Second, we introduce a correlation-enhanced emotion
conditioning module with learnable Emotion Banks that explicitly capture
inter-emotion relationships through vector quantization and attention-based
feature aggregation. Third, we design an emotion discrimination objective that
enforces affective consistency during the diffusion process through
latent-space classification. Extensive experiments on MEAD and HDTF datasets
demonstrate our method's superiority, outperforming state-of-the-art approaches
in emotion accuracy while maintaining competitive lip-sync performance.
Qualitative results and user studies further confirm our method's ability to
generate identity-preserving portraits with rich, correlated emotional
expressions that naturally adapt to unseen identities.