Textually Pretrained Speech Language Models
Speech language models (SpeechLMs) process and generate acoustic data only,
without textual supervision. In this work, we propose TWIST, a method for
training SpeechLMs using a warm-start from a pretrained textual language
models. We show using both automatic and human evaluations that TWIST
outperforms a cold-start SpeechLM across the board. We empirically analyze the
effect of different model design choices such as the speech tokenizer, the
pretrained textual model, and the dataset size. We find that model and dataset
scale both play an important role in constructing better-performing SpeechLMs.
Based on our observations, we present the largest (to the best of our
knowledge) SpeechLM both in terms of number of parameters and training data. We
additionally introduce two spoken versions of the StoryCloze textual benchmark
to further improve model evaluation and advance future research in the field.
Speech samples can be found on our website:
https://pages.cs.huji.ac.il/adiyoss-lab/twist/ .
646c282eb1202bc77c149334
Michael Hassid
646c282eb1202bc77c149335
Tal Remez
claimed_verified
2023-06-08T16:08:46.167Z
647f1b66838ac3601fc762d1
/avatars/24aaaeb700690bc84ad0212ce4ae9bd4.svg
Tal Remez
user
TalRemez
646c282eb1202bc77c149336
Tu Anh Nguyen
646c282eb1202bc77c149337
Itai Gat
claimed_verified
2023-05-23T08:11:22.030Z
62a8fa984d933c74bf410c16
/avatars/73519deba3176be9c23d49f749aee5da.svg
Itai Gat
user
itaigat
646c282eb1202bc77c149338
Alexis Conneau
admin_assigned
2023-05-23T08:47:24.758Z
61776a8f1e3e9ecbf77ddeea
/avatars/f2b9215fa02178194d0cbf2819681824.svg
Alexis C
user
aconneau
646c282eb1202bc77c149339
Felix Kreuk
admin_assigned
2023-05-23T09:42:05.792Z
645d0a4cf4f49de5809de159
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/IOWMd17Iwls0dXsY1OWjK.jpeg
Felix Kreuk
user
felixkreuk
646c282eb1202bc77c14933a
Jade Copet
646c282eb1202bc77c14933b
Alexandre Defossez
claimed_verified
2023-05-23T08:59:13.900Z
6357f5f50dce5559211a3503
https://cdn-avatars.huggingface.co/v1/production/uploads/1666708948380-noauth.jpeg
Alexandre Défossez
user
adefossez
646c282eb1202bc77c14933c
Gabriel Synnaeve
646c282eb1202bc77c14933d
Emmanuel Dupoux
646c282eb1202bc77c14933e
Roy Schwartz
646c282eb1202bc77c14933f
Yossi Adi
Textually Pretrained Speech Language Models
Speech language models (SpeechLMs) process and generate acoustic data only,
without textual supervision. In this work, we propose TWIST, a method for
training SpeechLMs using a warm-start from a pretrained textual language
models. We show using both automatic and human evaluations that TWIST
outperforms a cold-start SpeechLM across the board. We empirically analyze the
effect of different model design choices such as the speech tokenizer, the
pretrained textual model, and the dataset size. We find that model and dataset
scale both play an important role in constructing better-performing SpeechLMs.
Based on our observations, we present the largest (to the best of our
knowledge) SpeechLM both in terms of number of parameters and training data. We
additionally introduce two spoken versions of the StoryCloze textual benchmark
to further improve model evaluation and advance future research in the field.
Speech samples can be found on our website:
https://pages.cs.huji.ac.il/adiyoss-lab/twist/ .
TWIST improves SpeechLMs by leveraging pretrained textual language models, demonstrating superior performance across various evaluations and introducing new benchmarks for spoken data.
SpeechLMs warm-start pretrained textual language models speech tokenizer dataset size model scale dataset scale StoryCloze benchmark