AudioSlots: A slot-centric generative model for audio separation
In a range of recent works, object-centric architectures have been shown to
be suitable for unsupervised scene decomposition in the vision domain. Inspired
by these methods we present AudioSlots, a slot-centric generative model for
blind source separation in the audio domain. AudioSlots is built using
permutation-equivariant encoder and decoder networks. The encoder network based
on the Transformer architecture learns to map a mixed audio spectrogram to an
unordered set of independent source embeddings. The spatial broadcast decoder
network learns to generate the source spectrograms from the source embeddings.
We train the model in an end-to-end manner using a permutation invariant loss
function. Our results on Libri2Mix speech separation constitute a proof of
concept that this approach shows promise. We discuss the results and
limitations of our approach in detail, and further outline potential ways to
overcome the limitations and directions for future work.
645b1567e68c2d8346169965
Pradyumna Reddy
645b1567e68c2d8346169966
Scott Wisdom
645b1567e68c2d8346169967
Klaus Greff
645b1567e68c2d8346169968
John R. Hershey
645b1567e68c2d8346169969
Thomas Kipf
admin_assigned
2023-05-24T09:22:03.577Z
6413572b6cd62eb3ba2024e9
https://cdn-avatars.huggingface.co/v1/production/uploads/6413572b6cd62eb3ba2024e9/UCtMdeV8_N6w-SBnTDXBy.jpeg
Thomas Kipf
user
tkipf
AudioSlots: A slot-centric generative model for audio separation
In a range of recent works, object-centric architectures have been shown to
be suitable for unsupervised scene decomposition in the vision domain. Inspired
by these methods we present AudioSlots, a slot-centric generative model for
blind source separation in the audio domain. AudioSlots is built using
permutation-equivariant encoder and decoder networks. The encoder network based
on the Transformer architecture learns to map a mixed audio spectrogram to an
unordered set of independent source embeddings. The spatial broadcast decoder
network learns to generate the source spectrograms from the source embeddings.
We train the model in an end-to-end manner using a permutation invariant loss
function. Our results on Libri2Mix speech separation constitute a proof of
concept that this approach shows promise. We discuss the results and
limitations of our approach in detail, and further outline potential ways to
overcome the limitations and directions for future work.
AudioSlots, a slot-centric generative model based on permutation-equivariant networks, demonstrates potential for blind source separation using transformed audio spectrograms.
object-centric architectures slot-centric generative model blind source separation permutation-equivariant encoder permutation-equivariant decoder Transformer architecture unordered set source embeddings spatial broadcast decoder permutation invariant loss function Libri2Mix