Scaling Analysis of Interleaved Speech-Text Language Models
Existing Speech Language Model (SLM) scaling analysis paints a bleak picture.
They predict that SLMs require much more compute and data compared to text,
leading some to question the feasibility of training high-quality SLMs.
However, modern SLMs are often initialised from pre-trained TextLMs using
speech-text interleaving to allow knowledge transfer. This raises the question
- Do interleaved SLMs scale more efficiently than textless-SLMs? In this paper
we answer a resounding, yes! We conduct scaling analysis of interleaved SLMs by
training several dozen and analysing the scaling trends. We see that under this
setup SLMs scale more efficiently with compute. Additionally, our results
indicate that the scaling-dynamics are significantly different than
textless-SLMs, suggesting one should allocate notably more of the compute
budget for increasing model size over training tokens. We also study the role
of synthetic data and TextLM model families in unlocking this potential.
Results suggest, that our scaled up model achieves comparable performance with
leading models on speech semantic metrics while using less compute and data
than other approaches. We open source models, samples, and data -
https://pages.cs.huji.ac.il/adiyoss-lab/sims.
67ef63b5e8b932ae7a8d3043
Gallil Maimon
claimed_verified
2025-04-04T07:08:06.509Z
66b9bc2dacdbc1d0b39c3b50
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/hwR0pVfP_E8XjimXIxDOU.jpeg
Gallil Maimon
user
gallilmaimon
67ef63b5e8b932ae7a8d3044
Michael Hassid
admin_assigned
2025-04-04T07:24:39.934Z
6547411a9295970f878aa52e
/avatars/6e240f0add27bf1a6c04a9618eccdf83.svg
Michael Hassid
user
hassid
67ef63b5e8b932ae7a8d3045
Amit Roth
admin_assigned
2025-04-04T07:24:48.483Z
64b7b7b38ba7d6c922d753d6
https://cdn-avatars.huggingface.co/v1/production/uploads/64b7b7b38ba7d6c922d753d6/rt0thjYa84VZHy1BEcW4p.jpeg
Amit Roth
user
MajoRoth
67ef63b5e8b932ae7a8d3046
Yossi Adi
admin_assigned
2025-04-04T07:24:54.851Z
6481e135578646b5c2386728
https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
Yossi Adi
user
adiyoss
Scaling Analysis of Interleaved Speech-Text Language Models
Existing Speech Language Model (SLM) scaling analysis paints a bleak picture.
They predict that SLMs require much more compute and data compared to text,
leading some to question the feasibility of training high-quality SLMs.
However, modern SLMs are often initialised from pre-trained TextLMs using
speech-text interleaving to allow knowledge transfer. This raises the question
- Do interleaved SLMs scale more efficiently than textless-SLMs? In this paper
we answer a resounding, yes! We conduct scaling analysis of interleaved SLMs by
training several dozen and analysing the scaling trends. We see that under this
setup SLMs scale more efficiently with compute. Additionally, our results
indicate that the scaling-dynamics are significantly different than
textless-SLMs, suggesting one should allocate notably more of the compute
budget for increasing model size over training tokens. We also study the role
of synthetic data and TextLM model families in unlocking this potential.
Results suggest, that our scaled up model achieves comparable performance with
leading models on speech semantic metrics while using less compute and data
than other approaches. We open source models, samples, and data -
https://pages.cs.huji.ac.il/adiyoss-lab/sims.
Interleaved Speech Language Models scale more efficiently than textless models, achieving comparable performance with less compute and data.
Speech Language Model SLM TextLM speech-text interleaving knowledge transfer scaling analysis scaling trends synthetic data TextLM model families speech semantic metrics