AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation
In this study, we propose AniPortrait, a novel framework for generating
high-quality animation driven by audio and a reference portrait image. Our
methodology is divided into two stages. Initially, we extract 3D intermediate
representations from audio and project them into a sequence of 2D facial
landmarks. Subsequently, we employ a robust diffusion model, coupled with a
motion module, to convert the landmark sequence into photorealistic and
temporally consistent portrait animation. Experimental results demonstrate the
superiority of AniPortrait in terms of facial naturalness, pose diversity, and
visual quality, thereby offering an enhanced perceptual experience. Moreover,
our methodology exhibits considerable potential in terms of flexibility and
controllability, which can be effectively applied in areas such as facial
motion editing or face reenactment. We release code and model weights at
https://github.com/scutzzj/AniPortrait
6603ebf72a78438d282ebaca
Huawei Wei
6603ebf72a78438d282ebacb
Zejun Yang
6603ebf72a78438d282ebacc
Zhisheng Wang
AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation
In this study, we propose AniPortrait, a novel framework for generating
high-quality animation driven by audio and a reference portrait image. Our
methodology is divided into two stages. Initially, we extract 3D intermediate
representations from audio and project them into a sequence of 2D facial
landmarks. Subsequently, we employ a robust diffusion model, coupled with a
motion module, to convert the landmark sequence into photorealistic and
temporally consistent portrait animation. Experimental results demonstrate the
superiority of AniPortrait in terms of facial naturalness, pose diversity, and
visual quality, thereby offering an enhanced perceptual experience. Moreover,
our methodology exhibits considerable potential in terms of flexibility and
controllability, which can be effectively applied in areas such as facial
motion editing or face reenactment. We release code and model weights at
https://github.com/scutzzj/AniPortrait
AniPortrait generates high-quality portrait animations from audio and a reference image using 3D intermediate representations, 2D facial landmarks, and a diffusion model with a motion module.
3D intermediate representations 2D facial landmarks diffusion model motion module