OpenBEATs: A Fully Open-Source General-Purpose Audio Encoder
Masked token prediction has emerged as a powerful pre-training objective
across language, vision, and speech, offering the potential to unify these
diverse modalities through a single pre-training task. However, its application
for general audio understanding remains underexplored, with BEATs being the
only notable example. BEATs has seen limited modifications due to the absence
of open-source pre-training code. Furthermore, BEATs was trained only on
AudioSet, restricting its broader downstream applicability. To address these
gaps, we present OpenBEATs, an open-source framework that extends BEATs via
multi-domain audio pre-training. We conduct comprehensive evaluations across
six types of tasks, twenty five datasets, and three audio domains, including
audio reasoning tasks such as audio question answering, entailment, and
captioning. OpenBEATs achieves state-of-the-art performance on six bioacoustics
datasets, two environmental sound datasets and five reasoning datasets,
performing better than models exceeding a billion parameters at one-fourth
their parameter size. These results demonstrate the effectiveness of
multi-domain datasets and masked token prediction task to learn general-purpose
audio representations. To promote further research and reproducibility, we
release all pre-training and evaluation code, pretrained and fine-tuned
checkpoints, and training logs at https://shikhar-s.github.io/OpenBEATs
687ebeccf55f755d9b1f3ae0
Shikhar Bharadwaj
687ebeccf55f755d9b1f3ae1
Samuele Cornell
687ebeccf55f755d9b1f3ae2
Kwanghee Choi
687ebeccf55f755d9b1f3ae3
Satoru Fukayama
687ebeccf55f755d9b1f3ae4
Hye-jin Shim
687ebeccf55f755d9b1f3ae5
Soham Deshmukh
687ebeccf55f755d9b1f3ae6
Shinji Watanabe
OpenBEATs: A Fully Open-Source General-Purpose Audio Encoder
Masked token prediction has emerged as a powerful pre-training objective
across language, vision, and speech, offering the potential to unify these
diverse modalities through a single pre-training task. However, its application
for general audio understanding remains underexplored, with BEATs being the
only notable example. BEATs has seen limited modifications due to the absence
of open-source pre-training code. Furthermore, BEATs was trained only on
AudioSet, restricting its broader downstream applicability. To address these
gaps, we present OpenBEATs, an open-source framework that extends BEATs via
multi-domain audio pre-training. We conduct comprehensive evaluations across
six types of tasks, twenty five datasets, and three audio domains, including
audio reasoning tasks such as audio question answering, entailment, and
captioning. OpenBEATs achieves state-of-the-art performance on six bioacoustics
datasets, two environmental sound datasets and five reasoning datasets,
performing better than models exceeding a billion parameters at one-fourth
their parameter size. These results demonstrate the effectiveness of
multi-domain datasets and masked token prediction task to learn general-purpose
audio representations. To promote further research and reproducibility, we
release all pre-training and evaluation code, pretrained and fine-tuned
checkpoints, and training logs at https://shikhar-s.github.io/OpenBEATs
OpenBEATs, an open-source framework extending BEATs with multi-domain audio pre-training, achieves state-of-the-art performance on various audio tasks with a smaller parameter size compared to larger models.
masked token prediction pre-training AudioSet OpenBEATs multi-domain audio pre-training audio reasoning audio question answering entailment captioning bioacoustics datasets environmental sound datasets reasoning datasets parameter-efficient fine-tuning