DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for
  Single Image Talking Face Generation
The generation of emotional talking faces from a single portrait image
remains a significant challenge. The simultaneous achievement of expressive
emotional talking and accurate lip-sync is particularly difficult, as
expressiveness is often compromised for the accuracy of lip-sync. As widely
adopted by many prior works, the LSTM network often fails to capture the
subtleties and variations of emotional expressions. To address these
challenges, we introduce DREAM-Talk, a two-stage diffusion-based audio-driven
framework, tailored for generating diverse expressions and accurate lip-sync
concurrently. In the first stage, we propose EmoDiff, a novel diffusion module
that generates diverse highly dynamic emotional expressions and head poses in
accordance with the audio and the referenced emotion style. Given the strong
correlation between lip motion and audio, we then refine the dynamics with
enhanced lip-sync accuracy using audio features and emotion style. To this end,
we deploy a video-to-video rendering module to transfer the expressions and lip
motions from our proxy 3D avatar to an arbitrary portrait. Both quantitatively
and qualitatively, DREAM-Talk outperforms state-of-the-art methods in terms of
expressiveness, lip-sync accuracy and perceptual quality.
6584edf9e07563d421f639c2
Chenxu Zhang
6584edf9e07563d421f639c3
Chao Wang
6584edf9e07563d421f639c4
Jianfeng Zhang
6584edf9e07563d421f639c5
Hongyi Xu
6584edf9e07563d421f639c6
Guoxian Song
admin_assigned
2023-12-22T11:05:22.392Z
63086a237dc1b1a54cc6c24d
/avatars/477b94134edc4c18c8f769ecbb7d8091.svg
Song
user
guoxiansong
6584edf9e07563d421f639c7
You Xie
admin_assigned
2023-12-22T11:05:42.961Z
6408dfd4b6a334f53e24023c
/avatars/b7e3fa4fbec6313e94ff3384b74dabfc.svg
You Xie
user
youxie
6584edf9e07563d421f639c8
Linjie Luo
6584edf9e07563d421f639c9
Yapeng Tian
6584edf9e07563d421f639ca
Xiaohu Guo
6584edf9e07563d421f639cb
Jiashi Feng
DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for
  Single Image Talking Face Generation
The generation of emotional talking faces from a single portrait image
remains a significant challenge. The simultaneous achievement of expressive
emotional talking and accurate lip-sync is particularly difficult, as
expressiveness is often compromised for the accuracy of lip-sync. As widely
adopted by many prior works, the LSTM network often fails to capture the
subtleties and variations of emotional expressions. To address these
challenges, we introduce DREAM-Talk, a two-stage diffusion-based audio-driven
framework, tailored for generating diverse expressions and accurate lip-sync
concurrently. In the first stage, we propose EmoDiff, a novel diffusion module
that generates diverse highly dynamic emotional expressions and head poses in
accordance with the audio and the referenced emotion style. Given the strong
correlation between lip motion and audio, we then refine the dynamics with
enhanced lip-sync accuracy using audio features and emotion style. To this end,
we deploy a video-to-video rendering module to transfer the expressions and lip
motions from our proxy 3D avatar to an arbitrary portrait. Both quantitatively
and qualitatively, DREAM-Talk outperforms state-of-the-art methods in terms of
expressiveness, lip-sync accuracy and perceptual quality.
DREAM-Talk is a two-stage diffusion-based framework generating expressive emotional talking faces with accurate lip-sync from single portrait images.
LSTM network diffusion-based audio-driven framework EmoDiff diffusion module audio features video-to-video rendering module proxy 3D avatar