SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction
  Fine-Tuning
We introduce SIFT (Speech Instruction Fine-Tuning), a 50M-example dataset
designed for instruction fine-tuning and pre-training of speech-text large
language models (LLMs). SIFT-50M is built from publicly available speech
corpora, which collectively contain 14K hours of speech, and leverages LLMs
along with off-the-shelf expert models. The dataset spans five languages,
encompassing a diverse range of speech understanding as well as controllable
speech generation instructions. Using SIFT-50M, we train SIFT-LLM, which
outperforms existing speech-text LLMs on instruction-following benchmarks while
achieving competitive performance on foundational speech tasks. To support
further research, we also introduce EvalSIFT, a benchmark dataset specifically
designed to evaluate the instruction-following capabilities of speech-text
LLMs.
67fdbfcccaa65039e8c3d8ae
Prabhat Pandey
claimed_verified
2025-04-16T09:23:52.370Z
67dcd93f73e2178fe917b893
/avatars/86ec25d32c45da3937280ef9a619f19e.svg
Prabhat Pandey
user
panprabh
67fdbfcccaa65039e8c3d8af
Rupak Vignesh Swaminathan
67fdbfcccaa65039e8c3d8b0
K V Vijay Girish
claimed_verified
2025-04-16T09:23:49.809Z
66279810009f1bfdc6bf71bf
/avatars/97f561c91b92e1abb4fe6f0b5c688126.svg
Girish
user
vijaygirish2001
67fdbfcccaa65039e8c3d8b1
Arunasish Sen
claimed_verified
2025-04-16T11:58:01.767Z
66367dfb2d6b86ff193dbbe0
/avatars/0a0c230bb5fc81a28a166691146cf807.svg
Arunasish Sen
user
svinxz
67fdbfcccaa65039e8c3d8b2
Jian Xie
67fdbfcccaa65039e8c3d8b3
Grant P. Strimel
67fdbfcccaa65039e8c3d8b4
Andreas Schwarz
SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction
  Fine-Tuning
We introduce SIFT (Speech Instruction Fine-Tuning), a 50M-example dataset
designed for instruction fine-tuning and pre-training of speech-text large
language models (LLMs). SIFT-50M is built from publicly available speech
corpora, which collectively contain 14K hours of speech, and leverages LLMs
along with off-the-shelf expert models. The dataset spans five languages,
encompassing a diverse range of speech understanding as well as controllable
speech generation instructions. Using SIFT-50M, we train SIFT-LLM, which
outperforms existing speech-text LLMs on instruction-following benchmarks while
achieving competitive performance on foundational speech tasks. To support
further research, we also introduce EvalSIFT, a benchmark dataset specifically
designed to evaluate the instruction-following capabilities of speech-text
LLMs.
SIFT, a 50M-example speech-text dataset, fine-tunes and pre-trains LLMs for instruction-following and foundational speech tasks, outperforming existing models and using the EvalSIFT benchmark for evaluation.
SIFT instruction fine-tuning speech-text LLMs speech corpora LLMs SIFT-LLM EvalSIFT instruction-following benchmarks foundational speech tasks