MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation
Recent advances in video diffusion models have unlocked new potential for
realistic audio-driven talking video generation. However, achieving seamless
audio-lip synchronization, maintaining long-term identity consistency, and
producing natural, audio-aligned expressions in generated talking videos remain
significant challenges. To address these challenges, we propose Memory-guided
EMOtion-aware diffusion (MEMO), an end-to-end audio-driven portrait animation
approach to generate identity-consistent and expressive talking videos. Our
approach is built around two key modules: (1) a memory-guided temporal module,
which enhances long-term identity consistency and motion smoothness by
developing memory states to store information from a longer past context to
guide temporal modeling via linear attention; and (2) an emotion-aware audio
module, which replaces traditional cross attention with multi-modal attention
to enhance audio-video interaction, while detecting emotions from audio to
refine facial expressions via emotion adaptive layer norm. Extensive
quantitative and qualitative results demonstrate that MEMO generates more
realistic talking videos across diverse image and audio types, outperforming
state-of-the-art methods in overall quality, audio-lip synchronization,
identity consistency, and expression-emotion alignment.
675265f58be943d9cb551721
Longtao Zheng
claimed_verified
2025-01-24T10:01:44.427Z
63db5dc49f2687298a1547bf
https://cdn-avatars.huggingface.co/v1/production/uploads/63db5dc49f2687298a1547bf/xVFi0kRkYud191cQgma16.jpeg
Longtao Zheng
user
ltzheng
675265f58be943d9cb551722
Yifan Zhang
claimed_verified
2025-06-25T08:10:52.657Z
63aed0e7f873109b112dbb1b
https://cdn-avatars.huggingface.co/v1/production/uploads/63aed0e7f873109b112dbb1b/JkSkQ1a2SLq5eTTnz9F05.jpeg
Yifan Zhang
user
Vanint
675265f58be943d9cb551723
Hanzhong Guo
675265f58be943d9cb551724
Jiachun Pan
675265f58be943d9cb551725
Zhenxiong Tan
675265f58be943d9cb551726
Jiahao Lu
claimed_verified
2025-05-29T07:47:27.061Z
64d3abeb1058ae59738ba8ce
/avatars/4a9de3db835ade2c20f0d2de678e85c4.svg
jiahao lu
user
jiahao97
675265f58be943d9cb551727
Chuanxin Tang
675265f58be943d9cb551728
Bo An
675265f58be943d9cb551729
Shuicheng Yan
MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation
Recent advances in video diffusion models have unlocked new potential for
realistic audio-driven talking video generation. However, achieving seamless
audio-lip synchronization, maintaining long-term identity consistency, and
producing natural, audio-aligned expressions in generated talking videos remain
significant challenges. To address these challenges, we propose Memory-guided
EMOtion-aware diffusion (MEMO), an end-to-end audio-driven portrait animation
approach to generate identity-consistent and expressive talking videos. Our
approach is built around two key modules: (1) a memory-guided temporal module,
which enhances long-term identity consistency and motion smoothness by
developing memory states to store information from a longer past context to
guide temporal modeling via linear attention; and (2) an emotion-aware audio
module, which replaces traditional cross attention with multi-modal attention
to enhance audio-video interaction, while detecting emotions from audio to
refine facial expressions via emotion adaptive layer norm. Extensive
quantitative and qualitative results demonstrate that MEMO generates more
realistic talking videos across diverse image and audio types, outperforming
state-of-the-art methods in overall quality, audio-lip synchronization,
identity consistency, and expression-emotion alignment.
Memory-guided EMOtion-aware diffusion (MEMO) enhances the realism of audio-driven talking videos by ensuring audio-lip synchronization, long-term identity consistency, and natural expression alignment.
memory-guided temporal module linear attention emotion-aware audio module multi-modal attention emotion adaptive layer norm audio-lip synchronization identity consistency expression-emotion alignment