SonicVerse: Multi-Task Learning for Music Feature-Informed Captioning
Detailed captions that accurately reflect the characteristics of a music
piece can enrich music databases and drive forward research in music AI. This
paper introduces a multi-task music captioning model, SonicVerse, that
integrates caption generation with auxiliary music feature detection tasks such
as key detection, vocals detection, and more, so as to directly capture both
low-level acoustic details as well as high-level musical attributes. The key
contribution is a projection-based architecture that transforms audio input
into language tokens, while simultaneously detecting music features through
dedicated auxiliary heads. The outputs of these heads are also projected into
language tokens, to enhance the captioning input. This framework not only
produces rich, descriptive captions for short music fragments but also directly
enables the generation of detailed time-informed descriptions for longer music
pieces, by chaining the outputs using a large-language model. To train the
model, we extended the MusicBench dataset by annotating it with music features
using MIRFLEX, a modular music feature extractor, resulting in paired audio,
captions and music feature data. Experimental results show that incorporating
features in this way improves the quality and detail of the generated captions.
685393f499bf39f9665c79db
Anuradha Chopra
admin_assigned
2025-06-20T12:30:18.910Z
65fd1805883a1c3b4b1fde10
/avatars/ff475f757e190021f66e1f3c0fe5bd17.svg
Anuradha Chopra
user
annabeth97c
685393f499bf39f9665c79dc
Abhinaba Roy
685393f499bf39f9665c79dd
Dorien Herremans
admin_assigned
2025-06-20T12:30:29.316Z
655431b2997379e9b0999d23
https://cdn-avatars.huggingface.co/v1/production/uploads/655431b2997379e9b0999d23/OhHCPCXfOS61Z7CSewUCU.png
Dorien Herremans
user
dorienh
SonicVerse: Multi-Task Learning for Music Feature-Informed Captioning
Detailed captions that accurately reflect the characteristics of a music
piece can enrich music databases and drive forward research in music AI. This
paper introduces a multi-task music captioning model, SonicVerse, that
integrates caption generation with auxiliary music feature detection tasks such
as key detection, vocals detection, and more, so as to directly capture both
low-level acoustic details as well as high-level musical attributes. The key
contribution is a projection-based architecture that transforms audio input
into language tokens, while simultaneously detecting music features through
dedicated auxiliary heads. The outputs of these heads are also projected into
language tokens, to enhance the captioning input. This framework not only
produces rich, descriptive captions for short music fragments but also directly
enables the generation of detailed time-informed descriptions for longer music
pieces, by chaining the outputs using a large-language model. To train the
model, we extended the MusicBench dataset by annotating it with music features
using MIRFLEX, a modular music feature extractor, resulting in paired audio,
captions and music feature data. Experimental results show that incorporating
features in this way improves the quality and detail of the generated captions.
SonicVerse, a multi-task music captioning model, integrates audio feature detection to enhance caption quality and enable detailed descriptions of music pieces.
multi-task music captioning SonicVerse caption generation key detection vocals detection projection-based architecture language tokens auxiliary heads time-informed descriptions large-language model MusicBench dataset MIRFLEX music feature extractor