SoloAudio: Target Sound Extraction with Language-oriented Audio
  Diffusion Transformer
In this paper, we introduce SoloAudio, a novel diffusion-based generative
model for target sound extraction (TSE). Our approach trains latent diffusion
models on audio, replacing the previous U-Net backbone with a skip-connected
Transformer that operates on latent features. SoloAudio supports both
audio-oriented and language-oriented TSE by utilizing a CLAP model as the
feature extractor for target sounds. Furthermore, SoloAudio leverages synthetic
audio generated by state-of-the-art text-to-audio models for training,
demonstrating strong generalization to out-of-domain data and unseen sound
events. We evaluate this approach on the FSD Kaggle 2018 mixture dataset and
real data from AudioSet, where SoloAudio achieves the state-of-the-art results
on both in-domain and out-of-domain data, and exhibits impressive zero-shot and
few-shot capabilities. Source code and demos are released.
66eae4cfca4f15856bb1ceb4
Helin Wang
claimed_verified
2024-09-19T13:26:27.158Z
63ecfb5ec5b3c734085db9ed
/avatars/0b1d03dcd7997ad1daa764fb76f88993.svg
Helin Wang
user
westbrook
66eae4cfca4f15856bb1ceb5
Jiarui Hai
claimed_verified
2024-09-22T12:48:01.790Z
6363e9e4969bdae89e10b5ac
https://cdn-avatars.huggingface.co/v1/production/uploads/6363e9e4969bdae89e10b5ac/uR0jz8pBJIQPmKC90OJgI.jpeg
Jiarui Hai
user
Higobeatz
66eae4cfca4f15856bb1ceb6
Yen-Ju Lu
66eae4cfca4f15856bb1ceb7
Karan Thakkar
claimed_verified
2025-06-07T05:51:03.430Z
634452e00a284e1e9f1d3d9d
https://cdn-avatars.huggingface.co/v1/production/uploads/634452e00a284e1e9f1d3d9d/jlJXRNxg4OH-zT8z9oDrr.jpeg
Karan Thakkar
user
carankt
66eae4cfca4f15856bb1ceb8
Mounya Elhilali
66eae4cfca4f15856bb1ceb9
Najim Dehak
SoloAudio: Target Sound Extraction with Language-oriented Audio
  Diffusion Transformer
In this paper, we introduce SoloAudio, a novel diffusion-based generative
model for target sound extraction (TSE). Our approach trains latent diffusion
models on audio, replacing the previous U-Net backbone with a skip-connected
Transformer that operates on latent features. SoloAudio supports both
audio-oriented and language-oriented TSE by utilizing a CLAP model as the
feature extractor for target sounds. Furthermore, SoloAudio leverages synthetic
audio generated by state-of-the-art text-to-audio models for training,
demonstrating strong generalization to out-of-domain data and unseen sound
events. We evaluate this approach on the FSD Kaggle 2018 mixture dataset and
real data from AudioSet, where SoloAudio achieves the state-of-the-art results
on both in-domain and out-of-domain data, and exhibits impressive zero-shot and
few-shot capabilities. Source code and demos are released.
SoloAudio, a diffusion-based generative model using a Transformer architecture, achieves state-of-the-art results in target sound extraction, supporting both audio-oriented and language-oriented tasks with strong generalization.
diffusion-based generative model latent diffusion models skip-connected Transformer CLAP model feature extractor text-to-audio models zero-shot few-shot capabilities