ZeroSep: Separate Anything in Audio with Zero Training
Audio source separation is fundamental for machines to understand complex
acoustic environments and underpins numerous audio applications. Current
supervised deep learning approaches, while powerful, are limited by the need
for extensive, task-specific labeled data and struggle to generalize to the
immense variability and open-set nature of real-world acoustic scenes. Inspired
by the success of generative foundation models, we investigate whether
pre-trained text-guided audio diffusion models can overcome these limitations.
We make a surprising discovery: zero-shot source separation can be achieved
purely through a pre-trained text-guided audio diffusion model under the right
configuration. Our method, named ZeroSep, works by inverting the mixed audio
into the diffusion model's latent space and then using text conditioning to
guide the denoising process to recover individual sources. Without any
task-specific training or fine-tuning, ZeroSep repurposes the generative
diffusion model for a discriminative separation task and inherently supports
open-set scenarios through its rich textual priors. ZeroSep is compatible with
a variety of pre-trained text-guided audio diffusion backbones and delivers
strong separation performance on multiple separation benchmarks, surpassing
even supervised methods.
683924d1ac00da416df936dd
Chao Huang
683924d1ac00da416df936de
Yuesheng Ma
683924d1ac00da416df936df
Junxuan Huang
683924d1ac00da416df936e0
Susan Liang
683924d1ac00da416df936e1
Yunlong Tang
683924d1ac00da416df936e2
Jing Bi
683924d1ac00da416df936e3
Wenqiang Liu
683924d1ac00da416df936e4
Nima Mesgarani
683924d1ac00da416df936e5
Chenliang Xu
ZeroSep: Separate Anything in Audio with Zero Training
Audio source separation is fundamental for machines to understand complex
acoustic environments and underpins numerous audio applications. Current
supervised deep learning approaches, while powerful, are limited by the need
for extensive, task-specific labeled data and struggle to generalize to the
immense variability and open-set nature of real-world acoustic scenes. Inspired
by the success of generative foundation models, we investigate whether
pre-trained text-guided audio diffusion models can overcome these limitations.
We make a surprising discovery: zero-shot source separation can be achieved
purely through a pre-trained text-guided audio diffusion model under the right
configuration. Our method, named ZeroSep, works by inverting the mixed audio
into the diffusion model's latent space and then using text conditioning to
guide the denoising process to recover individual sources. Without any
task-specific training or fine-tuning, ZeroSep repurposes the generative
diffusion model for a discriminative separation task and inherently supports
open-set scenarios through its rich textual priors. ZeroSep is compatible with
a variety of pre-trained text-guided audio diffusion backbones and delivers
strong separation performance on multiple separation benchmarks, surpassing
even supervised methods.
ZeroSep, a text-guided audio diffusion model, achieves zero-shot source separation through pre-trained models and text conditioning, outperforming supervised methods on various benchmarks.
audio source separation supervised deep learning generative foundation models text-guided audio diffusion models zero-shot source separation latent space denoising process discriminative separation task textual priors open-set scenarios