Multitrack Music Transcription with a Time-Frequency Perceiver
Multitrack music transcription aims to transcribe a music audio input into
the musical notes of multiple instruments simultaneously. It is a very
challenging task that typically requires a more complex model to achieve
satisfactory result. In addition, prior works mostly focus on transcriptions of
regular instruments, however, neglecting vocals, which are usually the most
important signal source if present in a piece of music. In this paper, we
propose a novel deep neural network architecture, Perceiver TF, to model the
time-frequency representation of audio input for multitrack transcription.
Perceiver TF augments the Perceiver architecture by introducing a hierarchical
expansion with an additional Transformer layer to model temporal coherence.
Accordingly, our model inherits the benefits of Perceiver that posses better
scalability, allowing it to well handle transcriptions of many instruments in a
single model. In experiments, we train a Perceiver TF to model 12 instrument
classes as well as vocal in a multi-task learning manner. Our result
demonstrates that the proposed system outperforms the state-of-the-art
counterparts (e.g., MT3 and SpecTNT) on various public datasets.
64929c9415180cc729d95a37
Wei-Tsung Lu
64929c9415180cc729d95a38
Ju-Chiang Wang
64929c9415180cc729d95a39
Yun-Ning Hung
Multitrack Music Transcription with a Time-Frequency Perceiver
Multitrack music transcription aims to transcribe a music audio input into
the musical notes of multiple instruments simultaneously. It is a very
challenging task that typically requires a more complex model to achieve
satisfactory result. In addition, prior works mostly focus on transcriptions of
regular instruments, however, neglecting vocals, which are usually the most
important signal source if present in a piece of music. In this paper, we
propose a novel deep neural network architecture, Perceiver TF, to model the
time-frequency representation of audio input for multitrack transcription.
Perceiver TF augments the Perceiver architecture by introducing a hierarchical
expansion with an additional Transformer layer to model temporal coherence.
Accordingly, our model inherits the benefits of Perceiver that posses better
scalability, allowing it to well handle transcriptions of many instruments in a
single model. In experiments, we train a Perceiver TF to model 12 instrument
classes as well as vocal in a multi-task learning manner. Our result
demonstrates that the proposed system outperforms the state-of-the-art
counterparts (e.g., MT3 and SpecTNT) on various public datasets.
A novel deep neural network, Perceiver TF, improves multitrack music transcription by integrating a hierarchical expansion and Transformer layer into the Perceiver architecture, achieving better scalability and performance compared to existing methods.
deep neural network Perceiver TF time-frequency representation Perceiver architecture hierarchical expansion Transformer layer temporal coherence multi-task learning multitrack transcription state-of-the-art