Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis
We are interested in a novel task, namely low-resource text-to-talking
avatar. Given only a few-minute-long talking person video with the audio track
as the training data and arbitrary texts as the driving input, we aim to
synthesize high-quality talking portrait videos corresponding to the input
text. This task has broad application prospects in the digital human industry
but has not been technically achieved yet due to two challenges: (1) It is
challenging to mimic the timbre from out-of-domain audio for a traditional
multi-speaker Text-to-Speech system. (2) It is hard to render high-fidelity and
lip-synchronized talking avatars with limited training data. In this paper, we
introduce Adaptive Text-to-Talking Avatar (Ada-TTA), which (1) designs a
generic zero-shot multi-speaker TTS model that well disentangles the text
content, timbre, and prosody; and (2) embraces recent advances in neural
rendering to achieve realistic audio-driven talking face video generation. With
these designs, our method overcomes the aforementioned two challenges and
achieves to generate identity-preserving speech and realistic talking person
video. Experiments demonstrate that our method could synthesize realistic,
identity-preserving, and audio-visual synchronized talking avatar videos.
647fddff9466a3c9ba4ec5c8
Zhenhui Ye
claimed_verified
2023-06-07T19:39:59.177Z
6277fcd8c5c547c31b80cc77
/avatars/7dab3f5c879b5c5eb86f822119a9b5bb.svg
Zhenhui Ye
user
yerfor
647fddff9466a3c9ba4ec5c9
Ziyue Jiang
647fddff9466a3c9ba4ec5ca
Yi Ren
claimed_verified
2023-06-07T10:10:04.128Z
6188c6aa72735d93d362efc8
/avatars/35bdd5f859c7fa47062e8153f974c32c.svg
Yi Ren
user
RayeRen
647fddff9466a3c9ba4ec5cb
Jinglin Liu
admin_assigned
2023-06-12T14:05:49.821Z
6205c6ad00e630c7a5d0fc2e
https://cdn-avatars.huggingface.co/v1/production/uploads/1645184905991-6205c6ad00e630c7a5d0fc2e.jpeg
Jinglin Liu
user
Silentlin
647fddff9466a3c9ba4ec5cc
Chen Zhang
647fddff9466a3c9ba4ec5cd
Xiang Yin
647fddff9466a3c9ba4ec5ce
Zejun Ma
647fddff9466a3c9ba4ec5cf
Zhou Zhao
Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis
We are interested in a novel task, namely low-resource text-to-talking
avatar. Given only a few-minute-long talking person video with the audio track
as the training data and arbitrary texts as the driving input, we aim to
synthesize high-quality talking portrait videos corresponding to the input
text. This task has broad application prospects in the digital human industry
but has not been technically achieved yet due to two challenges: (1) It is
challenging to mimic the timbre from out-of-domain audio for a traditional
multi-speaker Text-to-Speech system. (2) It is hard to render high-fidelity and
lip-synchronized talking avatars with limited training data. In this paper, we
introduce Adaptive Text-to-Talking Avatar (Ada-TTA), which (1) designs a
generic zero-shot multi-speaker TTS model that well disentangles the text
content, timbre, and prosody; and (2) embraces recent advances in neural
rendering to achieve realistic audio-driven talking face video generation. With
these designs, our method overcomes the aforementioned two challenges and
achieves to generate identity-preserving speech and realistic talking person
video. Experiments demonstrate that our method could synthesize realistic,
identity-preserving, and audio-visual synchronized talking avatar videos.
Ada-TTA synthesizes realistic, identity-preserving, and synchronized talking avatars using a generic zero-shot multi-speaker TTS model and advances in neural rendering.
Text-to-Talking Avatar Adaptive Text-to-Talking Avatar zero-shot multi-speaker TTS model timbre prosody neural rendering audio-driven talking face video generation