FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking
  Portrait
With the rapid advancement of diffusion-based generative models, portrait
image animation has achieved remarkable results. However, it still faces
challenges in temporally consistent video generation and fast sampling due to
its iterative sampling nature. This paper presents FLOAT, an audio-driven
talking portrait video generation method based on flow matching generative
model. We shift the generative modeling from the pixel-based latent space to a
learned motion latent space, enabling efficient design of temporally consistent
motion. To achieve this, we introduce a transformer-based vector field
predictor with a simple yet effective frame-wise conditioning mechanism.
Additionally, our method supports speech-driven emotion enhancement, enabling a
natural incorporation of expressive motions. Extensive experiments demonstrate
that our method outperforms state-of-the-art audio-driven talking portrait
methods in terms of visual quality, motion fidelity, and efficiency.
674e9ca6068c8178f4815e4c
Taekyung Ki
claimed_verified
2024-12-04T09:46:17.846Z
66b57c77778c98d29446c8ec
/avatars/c176bb7c072f3093f6a0786c87d384d8.svg
Taekyung Ki
user
taekyungki
674e9ca6068c8178f4815e4d
Dongchan Min
admin_assigned
2024-12-03T15:24:09.885Z
663cb73c749d810d11ab7d32
/avatars/c95a9e07e23e16e3f362066f8478908e.svg
dongchan min
user
kevin0417
674e9ca6068c8178f4815e4e
Gyoungsu Chae
FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking
  Portrait
With the rapid advancement of diffusion-based generative models, portrait
image animation has achieved remarkable results. However, it still faces
challenges in temporally consistent video generation and fast sampling due to
its iterative sampling nature. This paper presents FLOAT, an audio-driven
talking portrait video generation method based on flow matching generative
model. We shift the generative modeling from the pixel-based latent space to a
learned motion latent space, enabling efficient design of temporally consistent
motion. To achieve this, we introduce a transformer-based vector field
predictor with a simple yet effective frame-wise conditioning mechanism.
Additionally, our method supports speech-driven emotion enhancement, enabling a
natural incorporation of expressive motions. Extensive experiments demonstrate
that our method outperforms state-of-the-art audio-driven talking portrait
methods in terms of visual quality, motion fidelity, and efficiency.
 FLOAT method generates high-quality, temporally consistent, and emotion-enhanced talking portraits using flow matching in a learned motion latent space with a transformer-based vector field predictor.
diffusion-based generative models flow matching generative model motion latent space transformer-based vector field predictor frame-wise conditioning mechanism