MulliVC: Multi-lingual Voice Conversion With Cycle Consistency
Voice conversion aims to modify the source speaker's voice to resemble the
target speaker while preserving the original speech content. Despite notable
advancements in voice conversion these days, multi-lingual voice conversion
(including both monolingual and cross-lingual scenarios) has yet to be
extensively studied. It faces two main challenges: 1) the considerable
variability in prosody and articulation habits across languages; and 2) the
rarity of paired multi-lingual datasets from the same speaker. In this paper,
we propose MulliVC, a novel voice conversion system that only converts timbre
and keeps original content and source language prosody without multi-lingual
paired data. Specifically, each training step of MulliVC contains three
substeps: In step one the model is trained with monolingual speech data; then,
steps two and three take inspiration from back translation, construct a
cyclical process to disentangle the timbre and other information (content,
prosody, and other language-related information) in the absence of
multi-lingual data from the same speaker. Both objective and subjective results
indicate that MulliVC significantly surpasses other methods in both monolingual
and cross-lingual contexts, demonstrating the system's efficacy and the
viability of the three-step approach with cycle consistency. Audio samples can
be found on our demo page (mullivc.github.io).
66b97e28a373e756edee2498
Jiawei Huang
admin_assigned
2024-08-13T11:03:31.829Z
63b2eaad677046a142862047
/avatars/c6d6b06e7d815ab4214dc690ae4b401b.svg
Jiawei Huang
user
aweiqifei
66b97e28a373e756edee2499
Chen Zhang
66b97e28a373e756edee249a
Yi Ren
66b97e28a373e756edee249b
Ziyue Jiang
66b97e28a373e756edee249c
Zhenhui Ye
admin_assigned
2024-08-13T11:09:57.255Z
6277fcd8c5c547c31b80cc77
/avatars/7dab3f5c879b5c5eb86f822119a9b5bb.svg
Zhenhui Ye
user
yerfor
66b97e28a373e756edee249d
Jinglin Liu
admin_assigned
2024-08-13T11:10:06.161Z
6205c6ad00e630c7a5d0fc2e
https://cdn-avatars.huggingface.co/v1/production/uploads/1645184905991-6205c6ad00e630c7a5d0fc2e.jpeg
Jinglin Liu
user
Silentlin
66b97e28a373e756edee249e
Jinzheng He
admin_assigned
2024-08-13T11:10:12.247Z
6594f06ac04427eb38444bce
/avatars/b13fbf589b25eff038deb3fa12d95871.svg
Jinzheng He
user
jinzheng-he
66b97e28a373e756edee249f
Xiang Yin
66b97e28a373e756edee24a0
Zhou Zhao
MulliVC: Multi-lingual Voice Conversion With Cycle Consistency
Voice conversion aims to modify the source speaker's voice to resemble the
target speaker while preserving the original speech content. Despite notable
advancements in voice conversion these days, multi-lingual voice conversion
(including both monolingual and cross-lingual scenarios) has yet to be
extensively studied. It faces two main challenges: 1) the considerable
variability in prosody and articulation habits across languages; and 2) the
rarity of paired multi-lingual datasets from the same speaker. In this paper,
we propose MulliVC, a novel voice conversion system that only converts timbre
and keeps original content and source language prosody without multi-lingual
paired data. Specifically, each training step of MulliVC contains three
substeps: In step one the model is trained with monolingual speech data; then,
steps two and three take inspiration from back translation, construct a
cyclical process to disentangle the timbre and other information (content,
prosody, and other language-related information) in the absence of
multi-lingual data from the same speaker. Both objective and subjective results
indicate that MulliVC significantly surpasses other methods in both monolingual
and cross-lingual contexts, demonstrating the system's efficacy and the
viability of the three-step approach with cycle consistency. Audio samples can
be found on our demo page (mullivc.github.io).
MulliVC is a novel voice conversion system that separately converts timbre and preserves content and prosody in monolingual and cross-lingual scenarios without requiring multi-lingual paired data.
voice conversion timbre prosody articulation multi-lingual back translation cyclical process cycle consistency