USAD: Universal Speech and Audio Representation via Distillation
Self-supervised learning (SSL) has revolutionized audio representations, yet
models often remain domain-specific, focusing on either speech or non-speech
tasks. In this work, we present Universal Speech and Audio Distillation (USAD),
a unified approach to audio representation learning that integrates diverse
audio types - speech, sound, and music - into a single model. USAD employs
efficient layer-to-layer distillation from domain-specific SSL models to train
a student on a comprehensive audio dataset. USAD offers competitive performance
across various benchmarks and datasets, including frame and instance-level
speech processing tasks, audio tagging, and sound classification, achieving
near state-of-the-art results with a single encoder on SUPERB and HEAR
benchmarks.
685a06460e4ad7e2197584c0
Heng-Jui Chang
claimed_verified
2025-06-24T08:08:49.104Z
6179f36a2a4e9edab3a95798
https://cdn-avatars.huggingface.co/v1/production/uploads/6179f36a2a4e9edab3a95798/0mmFY5lFzPC5k6_GSdmYQ.jpeg
Heng-Jui Chang
user
vectominist
685a06460e4ad7e2197584c1
Saurabhchand Bhati
claimed_verified
2025-06-25T08:09:27.586Z
67d301cbba86f5d66eb73d7c
/avatars/8546bbd2145c16d4be5675624516b649.svg
Saurabhchand Bhati
user
saurabhati
685a06460e4ad7e2197584c2
James Glass
685a06460e4ad7e2197584c3
Alexander H. Liu
USAD: Universal Speech and Audio Representation via Distillation
Self-supervised learning (SSL) has revolutionized audio representations, yet
models often remain domain-specific, focusing on either speech or non-speech
tasks. In this work, we present Universal Speech and Audio Distillation (USAD),
a unified approach to audio representation learning that integrates diverse
audio types - speech, sound, and music - into a single model. USAD employs
efficient layer-to-layer distillation from domain-specific SSL models to train
a student on a comprehensive audio dataset. USAD offers competitive performance
across various benchmarks and datasets, including frame and instance-level
speech processing tasks, audio tagging, and sound classification, achieving
near state-of-the-art results with a single encoder on SUPERB and HEAR
benchmarks.
USAD integrates diverse audio types using efficient layer-to-layer distillation from domain-specific models, achieving competitive performance across various benchmarks with a single encoder.
self-supervised learning universal speech and audio distillation domain-specific models layer-to-layer distillation frame and instance-level speech processing audio tagging sound classification encoder SUPERB benchmarks HEAR benchmarks