FantasyTalking2: Timestep-Layer Adaptive Preference Optimization for
  Audio-Driven Portrait Animation
Recent advances in audio-driven portrait animation have demonstrated
impressive capabilities. However, existing methods struggle to align with
fine-grained human preferences across multiple dimensions, such as motion
naturalness, lip-sync accuracy, and visual quality. This is due to the
difficulty of optimizing among competing preference objectives, which often
conflict with one another, and the scarcity of large-scale, high-quality
datasets with multidimensional preference annotations. To address these, we
first introduce Talking-Critic, a multimodal reward model that learns
human-aligned reward functions to quantify how well generated videos satisfy
multidimensional expectations. Leveraging this model, we curate Talking-NSQ, a
large-scale multidimensional human preference dataset containing 410K
preference pairs. Finally, we propose Timestep-Layer adaptive multi-expert
Preference Optimization (TLPO), a novel framework for aligning diffusion-based
portrait animation models with fine-grained, multidimensional preferences. TLPO
decouples preferences into specialized expert modules, which are then fused
across timesteps and network layers, enabling comprehensive, fine-grained
enhancement across all dimensions without mutual interference. Experiments
demonstrate that Talking-Critic significantly outperforms existing methods in
aligning with human preference ratings. Meanwhile, TLPO achieves substantial
improvements over baseline models in lip-sync accuracy, motion naturalness, and
visual quality, exhibiting superior performance in both qualitative and
quantitative evaluations. Ours project page:
https://fantasy-amap.github.io/fantasy-talking2/
68a2ccd3a4caabb4320e6473
MengChao Wang
68a2ccd3a4caabb4320e6474
Qiang Wang
claimed_verified
2025-08-18T06:53:09.368Z
653b195c5f1703225b2fd571
https://cdn-avatars.huggingface.co/v1/production/uploads/653b195c5f1703225b2fd571/aCDRKL6PBkzhh8-nzDiMo.jpeg
wangqiang
user
wangqiang9
68a2ccd3a4caabb4320e6475
Fan Jiang
68a2ccd3a4caabb4320e6476
Mu Xu
FantasyTalking2: Timestep-Layer Adaptive Preference Optimization for
  Audio-Driven Portrait Animation
Recent advances in audio-driven portrait animation have demonstrated
impressive capabilities. However, existing methods struggle to align with
fine-grained human preferences across multiple dimensions, such as motion
naturalness, lip-sync accuracy, and visual quality. This is due to the
difficulty of optimizing among competing preference objectives, which often
conflict with one another, and the scarcity of large-scale, high-quality
datasets with multidimensional preference annotations. To address these, we
first introduce Talking-Critic, a multimodal reward model that learns
human-aligned reward functions to quantify how well generated videos satisfy
multidimensional expectations. Leveraging this model, we curate Talking-NSQ, a
large-scale multidimensional human preference dataset containing 410K
preference pairs. Finally, we propose Timestep-Layer adaptive multi-expert
Preference Optimization (TLPO), a novel framework for aligning diffusion-based
portrait animation models with fine-grained, multidimensional preferences. TLPO
decouples preferences into specialized expert modules, which are then fused
across timesteps and network layers, enabling comprehensive, fine-grained
enhancement across all dimensions without mutual interference. Experiments
demonstrate that Talking-Critic significantly outperforms existing methods in
aligning with human preference ratings. Meanwhile, TLPO achieves substantial
improvements over baseline models in lip-sync accuracy, motion naturalness, and
visual quality, exhibiting superior performance in both qualitative and
quantitative evaluations. Ours project page:
https://fantasy-amap.github.io/fantasy-talking2/
A multimodal reward model and adaptive preference optimization framework improve audio-driven portrait animation by aligning with human preferences across multiple dimensions.
Talking-Critic multimodal reward model Talking-NSQ Timestep-Layer adaptive multi-expert Preference Optimization (TLPO) diffusion-based portrait animation models expert modules lip-sync accuracy motion naturalness visual quality