F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow
  Matching
This paper introduces F5-TTS, a fully non-autoregressive text-to-speech
system based on flow matching with Diffusion Transformer (DiT). Without
requiring complex designs such as duration model, text encoder, and phoneme
alignment, the text input is simply padded with filler tokens to the same
length as input speech, and then the denoising is performed for speech
generation, which was originally proved feasible by E2 TTS. However, the
original design of E2 TTS makes it hard to follow due to its slow convergence
and low robustness. To address these issues, we first model the input with
ConvNeXt to refine the text representation, making it easy to align with the
speech. We further propose an inference-time Sway Sampling strategy, which
significantly improves our model's performance and efficiency. This sampling
strategy for flow step can be easily applied to existing flow matching based
models without retraining. Our design allows faster training and achieves an
inference RTF of 0.15, which is greatly improved compared to state-of-the-art
diffusion-based TTS models. Trained on a public 100K hours multilingual
dataset, our Fairytaler Fakes Fluent and Faithful speech with Flow matching
(F5-TTS) exhibits highly natural and expressive zero-shot ability, seamless
code-switching capability, and speed control efficiency. Demo samples can be
found at https://SWivid.github.io/F5-TTS. We release all code and checkpoints
to promote community development.
670761c9bbae2dc3843a4164
Yushen Chen
admin_assigned
2024-10-10T09:54:29.316Z
636916bb85fef3ca96e2ef73
https://cdn-avatars.huggingface.co/v1/production/uploads/636916bb85fef3ca96e2ef73/waHubUG9Ck-EBID9PBnWZ.jpeg
Yushen CHEN
user
SWivid
670761c9bbae2dc3843a4165
Zhikang Niu
670761c9bbae2dc3843a4166
Ziyang Ma
670761c9bbae2dc3843a4167
Keqi Deng
admin_assigned
2024-10-10T09:55:02.556Z
6190e0a108a57f265f7f5246
/avatars/f7dc279ea9e922a2684dd23d4aa3f757.svg
Keqi Deng
user
D-Keqi
670761c9bbae2dc3843a4168
Chunhui Wang
admin_assigned
2024-10-10T09:55:07.827Z
66610c2ec629e0e089fc9cf9
/avatars/b3a958de94ba93a8b21ae6d1011f0b45.svg
Chun-Hui Wang
user
ChunHuiWangFN
670761c9bbae2dc3843a4169
Jian Zhao
670761c9bbae2dc3843a416a
Kai Yu
admin_assigned
2024-10-10T09:55:18.617Z
66c04955780d735f176eb72f
/avatars/10fb520fbaf028d80aa1fb17a1d17f06.svg
Kai Yu
user
kaiyu-hf
670761c9bbae2dc3843a416b
Xie Chen
F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow
  Matching
This paper introduces F5-TTS, a fully non-autoregressive text-to-speech
system based on flow matching with Diffusion Transformer (DiT). Without
requiring complex designs such as duration model, text encoder, and phoneme
alignment, the text input is simply padded with filler tokens to the same
length as input speech, and then the denoising is performed for speech
generation, which was originally proved feasible by E2 TTS. However, the
original design of E2 TTS makes it hard to follow due to its slow convergence
and low robustness. To address these issues, we first model the input with
ConvNeXt to refine the text representation, making it easy to align with the
speech. We further propose an inference-time Sway Sampling strategy, which
significantly improves our model's performance and efficiency. This sampling
strategy for flow step can be easily applied to existing flow matching based
models without retraining. Our design allows faster training and achieves an
inference RTF of 0.15, which is greatly improved compared to state-of-the-art
diffusion-based TTS models. Trained on a public 100K hours multilingual
dataset, our Fairytaler Fakes Fluent and Faithful speech with Flow matching
(F5-TTS) exhibits highly natural and expressive zero-shot ability, seamless
code-switching capability, and speed control efficiency. Demo samples can be
found at https://SWivid.github.io/F5-TTS. We release all code and checkpoints
to promote community development.
F5-TTS, a fully non-autoregressive text-to-speech system, improves E2 TTS by leveraging ConvNeXt and Sway Sampling for better performance and efficiency.
flow matching Diffusion Transformer (DiT) fully non-autoregressive denoising ConvNeXt inference-time Sway Sampling flow step diffusion-based TTS multilingual dataset zero-shot ability code-switching speed control