OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and
  Cleaning
The Open Whisper-style Speech Models (OWSM) project has developed a series of
fully open speech foundation models using academic-scale resources, but their
training data remains insufficient. This work enhances OWSM by integrating
YODAS, a large-scale web-crawled dataset with a Creative Commons license.
However, incorporating YODAS is nontrivial due to its wild nature, which
introduces challenges such as incorrect language labels and audio-text
misalignments. To address this, we develop a scalable data-cleaning pipeline
using public toolkits, yielding a dataset with 166,000 hours of speech across
75 languages. Our new series of OWSM v4 models, trained on this curated dataset
alongside existing OWSM data, significantly outperform previous versions on
multilingual benchmarks. Our models even match or surpass frontier industrial
models like Whisper and MMS in multiple scenarios. We will publicly release the
cleaned YODAS data, pre-trained models, and all associated scripts via the
ESPnet toolkit.
683e64eb3e5a54d05365ddc6
Yifan Peng
claimed_verified
2025-06-03T08:46:09.029Z
61809f31a367a8f5351ef353
https://cdn-avatars.huggingface.co/v1/production/uploads/61809f31a367a8f5351ef353/s5eQ00YeoirakzE_rJ0cy.jpeg
Yifan Peng
user
pyf98
683e64eb3e5a54d05365ddc7
Shakeel Muhammad
claimed_verified
2025-06-04T09:03:08.892Z
657821170766f274f3c9b35f
/avatars/3e95b1fb9578ec27d9ad176c34b979c4.svg
Muhammad Shakeel
user
shakeel1
683e64eb3e5a54d05365ddc8
Yui Sudo
683e64eb3e5a54d05365ddc9
William Chen
683e64eb3e5a54d05365ddca
Jinchuan Tian
683e64eb3e5a54d05365ddcb
Chyi-Jiunn Lin
683e64eb3e5a54d05365ddcc
Shinji Watanabe
OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and
  Cleaning
The Open Whisper-style Speech Models (OWSM) project has developed a series of
fully open speech foundation models using academic-scale resources, but their
training data remains insufficient. This work enhances OWSM by integrating
YODAS, a large-scale web-crawled dataset with a Creative Commons license.
However, incorporating YODAS is nontrivial due to its wild nature, which
introduces challenges such as incorrect language labels and audio-text
misalignments. To address this, we develop a scalable data-cleaning pipeline
using public toolkits, yielding a dataset with 166,000 hours of speech across
75 languages. Our new series of OWSM v4 models, trained on this curated dataset
alongside existing OWSM data, significantly outperform previous versions on
multilingual benchmarks. Our models even match or surpass frontier industrial
models like Whisper and MMS in multiple scenarios. We will publicly release the
cleaned YODAS data, pre-trained models, and all associated scripts via the
ESPnet toolkit.
The OWSM project is enhanced with a large-scale, cleaned web dataset, leading to improved multilingual speech models comparable to leading industrial models.
speach foundation models YODAS data-cleaning pipeline multilingual benchmarks Whisper MMS ESPnet toolkit