SpiRit-LM: Interleaved Spoken and Written Language Model
We introduce SPIRIT-LM, a foundation multimodal language model that freely
mixes text and speech. Our model is based on a pretrained text language model
that we extend to the speech modality by continuously training it on text and
speech units. Speech and text sequences are concatenated as a single set of
tokens, and trained with a word-level interleaving method using a small
automatically-curated speech-text parallel corpus. SPIRIT-LM comes in two
versions: a BASE version that uses speech semantic units and an EXPRESSIVE
version that models expressivity using pitch and style units in addition to the
semantic units. For both versions, the text is encoded with subword BPE tokens.
The resulting model displays both the semantic abilities of text models and the
expressive abilities of speech models. Additionally, we demonstrate that
SPIRIT-LM is able to learn new tasks in a few-shot fashion across modalities
(i.e. ASR, TTS, Speech Classification).
65c5984702262fe3b4fb9c57
Tu Anh Nguyen
admin_assigned
2024-02-09T10:17:00.027Z
62e7b942c9e83245ea5e14ec
/avatars/3223dae7ca6b80e6dc780c1b002b1524.svg
Tu Anh Nguyen
user
ntuanh
65c5984702262fe3b4fb9c58
Benjamin Muller
admin_assigned
2024-02-09T10:17:06.496Z
5e6b7153d4cd9779932a7600
/avatars/c1304f100758c5e2a54fd4d47f638841.svg
Benjamin Muller
user
benjamin-mlr
65c5984702262fe3b4fb9c59
Bokai Yu
admin_assigned
2024-02-09T10:17:26.496Z
65b3c0ee05c25412bb6aab2c
/avatars/2b3be41c899a29e2a0954dd7cd6cd5ab.svg
Bokai Yu
user
bokaiyu
65c5984702262fe3b4fb9c5a
Marta R. Costa-jussa
65c5984702262fe3b4fb9c5b
Maha Elbayad
admin_assigned
2024-02-09T10:17:42.732Z
64e243dff8934329dd72098f
https://cdn-avatars.huggingface.co/v1/production/uploads/64e243dff8934329dd72098f/JbB4zvlendHSRXjx_iC0j.jpeg
Maha Elbayad
user
elbayadm
65c5984702262fe3b4fb9c5c
Sravya Popuri
admin_assigned
2024-02-09T10:17:48.409Z
62e190f3926f4892a4c8099e
/avatars/2168f769c526362f9ed35473f4b30047.svg
Sravya Popuri
user
sravyapopuri388
65c5984702262fe3b4fb9c5d
Paul-Ambroise Duquenne
admin_assigned
2024-02-09T10:17:54.553Z
6331a09fe092098b57bc1d83
/avatars/cdbc4be17c160b5e1f957b6c1ae29e22.svg
Paul-Ambroise Duquenne
user
padqn
65c5984702262fe3b4fb9c5e
Robin Algayres
admin_assigned
2024-02-09T10:18:00.654Z
64d25f2035577e62a9df31a8
/avatars/fdaeb527f084d21e35de5a625c3629d6.svg
robin algayres
user
robinalgayres
65c5984702262fe3b4fb9c5f
Ruslan Mavlyutov
admin_assigned
2024-02-09T10:18:06.330Z
64dcfecb249785efad9a514d
/avatars/77de68494021f298bf2561b2ea9ffef8.svg
Ruslan Mavlyutov
user
mavlyutovrus
65c5984702262fe3b4fb9c60
Itai Gat
admin_assigned
2024-02-09T10:18:12.797Z
62a8fa984d933c74bf410c16
/avatars/73519deba3176be9c23d49f749aee5da.svg
Itai Gat
user
itaigat
65c5984702262fe3b4fb9c61
Gabriel Synnaeve
admin_assigned
2024-02-09T10:18:19.465Z
630eac7931970d1cd4fbacf2
/avatars/b7ccbddfa745db854dc342be1327cd53.svg
Gabriel Synnaeve
user
gsynnaeve
65c5984702262fe3b4fb9c62
Juan Pino
admin_assigned
2024-02-09T10:18:25.364Z
634deff6be5a827d4874546a
/avatars/05e0ad93d9c8c7e572c6f73feb649aeb.svg
Juan Pino
user
juanpino
65c5984702262fe3b4fb9c63
Benoit Sagot
65c5984702262fe3b4fb9c64
Emmanuel Dupoux
admin_assigned
2024-02-09T10:18:34.382Z
63317d2118711776b4663c3a
/avatars/7dedd1934c1000b6f81a2a37ec348347.svg
Emmanuel Dupoux
user
edupoux
SpiRit-LM: Interleaved Spoken and Written Language Model
We introduce SPIRIT-LM, a foundation multimodal language model that freely
mixes text and speech. Our model is based on a pretrained text language model
that we extend to the speech modality by continuously training it on text and
speech units. Speech and text sequences are concatenated as a single set of
tokens, and trained with a word-level interleaving method using a small
automatically-curated speech-text parallel corpus. SPIRIT-LM comes in two
versions: a BASE version that uses speech semantic units and an EXPRESSIVE
version that models expressivity using pitch and style units in addition to the
semantic units. For both versions, the text is encoded with subword BPE tokens.
The resulting model displays both the semantic abilities of text models and the
expressive abilities of speech models. Additionally, we demonstrate that
SPIRIT-LM is able to learn new tasks in a few-shot fashion across modalities
(i.e. ASR, TTS, Speech Classification).
A foundation multimodal language model combines text and speech using speech and text units, supporting semantic and expressive capabilities and few-shot learning across tasks.
multimodal language model speech semantic units subword BPE tokens expressivity pitch style units few-shot learning ASR TTS Speech Classification