Towards Robust Speech Representation Learning for Thousands of Languages
Self-supervised learning (SSL) has helped extend speech technologies to more
languages by reducing the need for labeled data. However, models are still far
from supporting the world's 7000+ languages. We propose XEUS, a Cross-lingual
Encoder for Universal Speech, trained on over 1 million hours of data across
4057 languages, extending the language coverage of SSL models 4-fold. We
combine 1 million hours of speech from existing publicly accessible corpora
with a newly created corpus of 7400+ hours from 4057 languages, which will be
publicly released. To handle the diverse conditions of multilingual speech
data, we augment the typical SSL masked prediction approach with a novel
dereverberation objective, increasing robustness. We evaluate XEUS on several
benchmarks, and show that it consistently outperforms or achieves comparable
results to state-of-the-art (SOTA) SSL models across a variety of tasks. XEUS
sets a new SOTA on the ML-SUPERB benchmark: it outperforms MMS 1B and w2v-BERT
2.0 v2 by 0.8% and 4.4% respectively, despite having less parameters or
pre-training data. Checkpoints, code, and data are found in
https://www.wavlab.org/activities/2024/xeus/.
668390c4c7d9ff0922172a12
William Chen
extracted_pending
2024-07-02T05:31:49.168Z
630438615c70c21d0eae6613
https://cdn-avatars.huggingface.co/v1/production/uploads/630438615c70c21d0eae6613/KEHsB-znDIKHt7M0eyP8j.png
William Chen
user
wanchichen
668390c4c7d9ff0922172a13
Wangyou Zhang
admin_assigned
2024-07-03T15:20:38.342Z
6205e8ab6c4c35a0a2415c81
https://cdn-avatars.huggingface.co/v1/production/uploads/1644558391919-6205e8ab6c4c35a0a2415c81.png
Wangyou Zhang
user
wyz
668390c4c7d9ff0922172a14
Yifan Peng
admin_assigned
2024-07-03T15:20:44.860Z
61809f31a367a8f5351ef353
https://cdn-avatars.huggingface.co/v1/production/uploads/61809f31a367a8f5351ef353/s5eQ00YeoirakzE_rJ0cy.jpeg
Yifan Peng
user
pyf98
668390c4c7d9ff0922172a15
Xinjian Li
admin_assigned
2024-07-03T15:20:57.432Z
648cd0277f7821f063c16e9a
/avatars/cb3d7745f22010d8a3f1b6a28975a561.svg
Xinjian Li
user
xinjianl
668390c4c7d9ff0922172a16
Jinchuan Tian
admin_assigned
2024-07-03T15:21:03.671Z
64770ff6d7cf39f2e937b882
/avatars/7b729501856f5073cf9e484811012f2a.svg
Jinchuan Tian
user
JinchuanTian
668390c4c7d9ff0922172a17
Jiatong Shi
admin_assigned
2024-07-03T15:21:09.544Z
6136ec7a40e43b8f748a0832
/avatars/089afa82ecfe6432d2dbfe81048c6ffa.svg
Jiatong Shi
user
ftshijt
668390c4c7d9ff0922172a18
Xuankai Chang
admin_assigned
2024-07-03T15:21:15.845Z
6136b5707c6d9b8e41469bec
/avatars/5b7f4ee0ba36bb6b87accda0ee27bb17.svg
Xuankai Chang
user
simpleoier
668390c4c7d9ff0922172a19
Soumi Maiti
admin_assigned
2024-07-03T15:21:21.415Z
6331d6465102481d8dd2b55e
/avatars/6dc792023b896e526d636c9c5d6e6547.svg
Soumi Maiti
user
soumi-maiti
668390c4c7d9ff0922172a1a
Karen Livescu
668390c4c7d9ff0922172a1b
Shinji Watanabe
admin_assigned
2024-07-03T15:21:47.692Z
5feb9dbc29bcc7d2abb1cb76
/avatars/908d392634377e674ba9e5a4318b101e.svg
Shinji Watanabe
user
sw005320
Towards Robust Speech Representation Learning for Thousands of Languages
Self-supervised learning (SSL) has helped extend speech technologies to more
languages by reducing the need for labeled data. However, models are still far
from supporting the world's 7000+ languages. We propose XEUS, a Cross-lingual
Encoder for Universal Speech, trained on over 1 million hours of data across
4057 languages, extending the language coverage of SSL models 4-fold. We
combine 1 million hours of speech from existing publicly accessible corpora
with a newly created corpus of 7400+ hours from 4057 languages, which will be
publicly released. To handle the diverse conditions of multilingual speech
data, we augment the typical SSL masked prediction approach with a novel
dereverberation objective, increasing robustness. We evaluate XEUS on several
benchmarks, and show that it consistently outperforms or achieves comparable
results to state-of-the-art (SOTA) SSL models across a variety of tasks. XEUS
sets a new SOTA on the ML-SUPERB benchmark: it outperforms MMS 1B and w2v-BERT
2.0 v2 by 0.8% and 4.4% respectively, despite having less parameters or
pre-training data. Checkpoints, code, and data are found in
https://www.wavlab.org/activities/2024/xeus/.
XEUS, a cross-lingual encoder for universal speech, enhances language coverage and robustness through novel dereverberation, outperforming existing models on various benchmarks with less data and parameters.
self-supervised learning cross-lingual encoder SSL masked prediction dereverberation language coverage multilingual speech data ML-SUPERB benchmark MMS 1B w2v-BERT 2.0 v2