WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words
Training on multiple modalities of input can augment the capabilities of a
language model. Here, we ask whether such a training regime can improve the
quality and efficiency of these systems as well. We focus on text--audio and
introduce Whisbert, which is inspired by the text--image approach of FLAVA
singh_flava_2022. In accordance with Babylm warstadt2023papers
guidelines, we pretrain Whisbert on a dataset comprising only 100 million words
plus their corresponding speech from the word-aligned version of the People's
Speech dataset galvez_peoples_2021. To assess the impact of
multimodality, we compare versions of the model that are trained on text only
and on both audio and text simultaneously. We find that while Whisbert is able
to perform well on multimodal masked modeling and surpasses the Babylm
baselines in most benchmark tasks, it struggles to optimize its complex
objective and outperform its text-only Whisbert baseline.
656fd6d77f50602340bea793
Lukas Wolf
claimed_verified
2023-12-07T08:30:04.253Z
63ea0c6e2e476376eb08b9fe
/avatars/a92b5261bd090f142945b60bdba19714.svg
Lukas 
user
luwo
656fd6d77f50602340bea794
Klemen Kotar
admin_assigned
2023-12-06T13:11:05.340Z
646d3cae4220471ca0c6cb13
/avatars/30e47db03a23d49d946cc5248b28959a.svg
Klemen Kotar
user
klemenk
656fd6d77f50602340bea795
Greta Tuckute
extracted_confirmed
2023-12-06T02:18:43.200Z
605a5f796ce6cabbb3474b55
/avatars/971866aa240126e480bbe01eff64c009.svg
Greta Tuckute
user
gretatuckute
656fd6d77f50602340bea796
Eghbal Hosseini
extracted_pending
2023-12-06T02:05:11.794Z
61de0406c59d602a3a85b186
/avatars/b95a77d962f7219bea7a397b272af792.svg
Eghbal Hosseini
user
ehoseini
656fd6d77f50602340bea797
Tamar Regev
656fd6d77f50602340bea798
Ethan Wilcox
656fd6d77f50602340bea799
Alex Warstadt
admin_assigned
2023-12-06T13:11:41.067Z
5ee772b8464d0272c8b24560
/avatars/81dfd862d13696936ed05e8ad966f926.svg
Alex Warstadt
user
alexwarstadt
WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words
Training on multiple modalities of input can augment the capabilities of a
language model. Here, we ask whether such a training regime can improve the
quality and efficiency of these systems as well. We focus on text--audio and
introduce Whisbert, which is inspired by the text--image approach of FLAVA
singh_flava_2022. In accordance with Babylm warstadt2023papers
guidelines, we pretrain Whisbert on a dataset comprising only 100 million words
plus their corresponding speech from the word-aligned version of the People's
Speech dataset galvez_peoples_2021. To assess the impact of
multimodality, we compare versions of the model that are trained on text only
and on both audio and text simultaneously. We find that while Whisbert is able
to perform well on multimodal masked modeling and surpasses the Babylm
baselines in most benchmark tasks, it struggles to optimize its complex
objective and outperform its text-only Whisbert baseline.
A multimodal text--audio language model, Whisbert, based on the FLAVA approach, shows improved performance on multimodal tasks but does not consistently outperform its text-only counterpart.
multimodality text--audio Whisbert FLAVA Babylm pretraining People's Speech dataset multimodal masked modeling