TalkVid: A Large-Scale Diversified Dataset for Audio-Driven Talking Head
  Synthesis
Audio-driven talking head synthesis has achieved remarkable photorealism, yet
state-of-the-art (SOTA) models exhibit a critical failure: they lack
generalization to the full spectrum of human diversity in ethnicity, language,
and age groups. We argue that this generalization gap is a direct symptom of
limitations in existing training data, which lack the necessary scale, quality,
and diversity. To address this challenge, we introduce TalkVid, a new
large-scale, high-quality, and diverse dataset containing 1244 hours of video
from 7729 unique speakers. TalkVid is curated through a principled, multi-stage
automated pipeline that rigorously filters for motion stability, aesthetic
quality, and facial detail, and is validated against human judgments to ensure
its reliability. Furthermore, we construct and release TalkVid-Bench, a
stratified evaluation set of 500 clips meticulously balanced across key
demographic and linguistic axes. Our experiments demonstrate that a model
trained on TalkVid outperforms counterparts trained on previous datasets,
exhibiting superior cross-dataset generalization. Crucially, our analysis on
TalkVid-Bench reveals performance disparities across subgroups that are
obscured by traditional aggregate metrics, underscoring its necessity for
future research. Code and data can be found in
https://github.com/FreedomIntelligence/TalkVid
68b510b7851c6e7b001eca5a
Shunian Chen
68b510b7851c6e7b001eca5b
Hejin Huang
68b510b7851c6e7b001eca5c
Yexin Liu
68b510b7851c6e7b001eca5d
Zihan Ye
68b510b7851c6e7b001eca5e
Pengcheng Chen
68b510b7851c6e7b001eca5f
Chenghao Zhu
68b510b7851c6e7b001eca60
Michael Guan
68b510b7851c6e7b001eca61
Rongsheng Wang
claimed_verified
2025-09-04T08:44:23.578Z
63ca949b04c979828315389d
https://cdn-avatars.huggingface.co/v1/production/uploads/63ca949b04c979828315389d/HS5xWNAYjjHeyAAwWJ11l.jpeg
wangrongsheng
user
wangrongsheng
68b510b7851c6e7b001eca62
Junying Chen
68b510b7851c6e7b001eca63
Guanbin Li
68b510b7851c6e7b001eca64
Ser-Nam Lim
68b510b7851c6e7b001eca65
Harry Yang
68b510b7851c6e7b001eca66
Benyou Wang
TalkVid: A Large-Scale Diversified Dataset for Audio-Driven Talking Head
  Synthesis
Audio-driven talking head synthesis has achieved remarkable photorealism, yet
state-of-the-art (SOTA) models exhibit a critical failure: they lack
generalization to the full spectrum of human diversity in ethnicity, language,
and age groups. We argue that this generalization gap is a direct symptom of
limitations in existing training data, which lack the necessary scale, quality,
and diversity. To address this challenge, we introduce TalkVid, a new
large-scale, high-quality, and diverse dataset containing 1244 hours of video
from 7729 unique speakers. TalkVid is curated through a principled, multi-stage
automated pipeline that rigorously filters for motion stability, aesthetic
quality, and facial detail, and is validated against human judgments to ensure
its reliability. Furthermore, we construct and release TalkVid-Bench, a
stratified evaluation set of 500 clips meticulously balanced across key
demographic and linguistic axes. Our experiments demonstrate that a model
trained on TalkVid outperforms counterparts trained on previous datasets,
exhibiting superior cross-dataset generalization. Crucially, our analysis on
TalkVid-Bench reveals performance disparities across subgroups that are
obscured by traditional aggregate metrics, underscoring its necessity for
future research. Code and data can be found in
https://github.com/FreedomIntelligence/TalkVid
TalkVid, a large-scale, high-quality, and diverse dataset, improves audio-driven talking head synthesis by enhancing generalization across human diversity and revealing subgroup performance disparities.
audio-driven talking head synthesis photorealism generalization training data large-scale dataset high-quality dataset diverse dataset motion stability aesthetic quality facial detail stratified evaluation set cross-dataset generalization performance disparities aggregate metrics