EmoKnob: Enhance Voice Cloning with Fine-Grained Emotion Control
While recent advances in Text-to-Speech (TTS) technology produce natural and
expressive speech, they lack the option for users to select emotion and control
intensity. We propose EmoKnob, a framework that allows fine-grained emotion
control in speech synthesis with few-shot demonstrative samples of arbitrary
emotion. Our framework leverages the expressive speaker representation space
made possible by recent advances in foundation voice cloning models. Based on
the few-shot capability of our emotion control framework, we propose two
methods to apply emotion control on emotions described by open-ended text,
enabling an intuitive interface for controlling a diverse array of nuanced
emotions. To facilitate a more systematic emotional speech synthesis field, we
introduce a set of evaluation metrics designed to rigorously assess the
faithfulness and recognizability of emotion control frameworks. Through
objective and subjective evaluations, we show that our emotion control
framework effectively embeds emotions into speech and surpasses emotion
expressiveness of commercial TTS services.
66fed130001816b29e7429ff
Haozhe Chen
extracted_confirmed
2024-10-03T17:31:25.648Z
6329cbaf74bc1990129c04e0
/avatars/8977d458af6aa2fa84d722e3cb7aeba1.svg
Tony Chen
user
tonychenxyz
66fed130001816b29e742a00
Run Chen
66fed130001816b29e742a01
Julia Hirschberg
EmoKnob: Enhance Voice Cloning with Fine-Grained Emotion Control
While recent advances in Text-to-Speech (TTS) technology produce natural and
expressive speech, they lack the option for users to select emotion and control
intensity. We propose EmoKnob, a framework that allows fine-grained emotion
control in speech synthesis with few-shot demonstrative samples of arbitrary
emotion. Our framework leverages the expressive speaker representation space
made possible by recent advances in foundation voice cloning models. Based on
the few-shot capability of our emotion control framework, we propose two
methods to apply emotion control on emotions described by open-ended text,
enabling an intuitive interface for controlling a diverse array of nuanced
emotions. To facilitate a more systematic emotional speech synthesis field, we
introduce a set of evaluation metrics designed to rigorously assess the
faithfulness and recognizability of emotion control frameworks. Through
objective and subjective evaluations, we show that our emotion control
framework effectively embeds emotions into speech and surpasses emotion
expressiveness of commercial TTS services.
A framework called EmoKnob enables fine-grained emotion control in speech synthesis using few-shot samples, surpassing commercial TTS services in emotion expressiveness.
Text-to-Speech (TTS) expressive speaker representation foundation voice cloning models emotion control framework intuitive interface evaluation metrics faithfulness recognizability