Futga: Towards Fine-grained Music Understanding through
  Temporally-enhanced Generative Augmentation
Existing music captioning methods are limited to generating concise global
descriptions of short music clips, which fail to capture fine-grained musical
characteristics and time-aware musical changes. To address these limitations,
we propose FUTGA, a model equipped with fined-grained music understanding
capabilities through learning from generative augmentation with temporal
compositions. We leverage existing music caption datasets and large language
models (LLMs) to synthesize fine-grained music captions with structural
descriptions and time boundaries for full-length songs. Augmented by the
proposed synthetic dataset, FUTGA is enabled to identify the music's temporal
changes at key transition points and their musical functions, as well as
generate detailed descriptions for each music segment. We further introduce a
full-length music caption dataset generated by FUTGA, as the augmentation of
the MusicCaps and the Song Describer datasets. We evaluate the automatically
generated captions on several downstream tasks, including music generation and
retrieval. The experiments demonstrate the quality of the generated captions
and the better performance in various downstream tasks achieved by the proposed
music captioning approach. Our code and datasets can be found in
https://huggingface.co/JoshuaW1997/FUTGA{blue{https://huggingface.co/JoshuaW1997/FUTGA}}.
66a9ae6f35070a5a52a2419e
Junda Wu
admin_assigned
2024-07-31T08:26:14.965Z
616c0cd640e2f69baa1c7adf
/avatars/9ad6ada73b3fb2d5fb696ec91f8edb1b.svg
Junda Wu
user
JoshuaW1997
66a9ae6f35070a5a52a2419f
Zachary Novack
claimed_verified
2024-07-31T07:44:04.364Z
643060c6cb3fe707b24c53a2
https://cdn-avatars.huggingface.co/v1/production/uploads/643060c6cb3fe707b24c53a2/MIoM9hrX0vV4XRyrm-4Kz.jpeg
Zachary Novack
user
ZacharyNovack
66a9ae6f35070a5a52a241a0
Amit Namburi
admin_assigned
2024-07-31T08:26:22.399Z
6611fb5555f5219b33a35856
/avatars/4ed9acc339d15acd8a5ce821acdac025.svg
Amit Namburi
user
namburiamit
66a9ae6f35070a5a52a241a1
Jiaheng Dai
66a9ae6f35070a5a52a241a2
Hao-Wen Dong
admin_assigned
2024-07-31T08:26:52.860Z
649cb9456209253a76854798
https://cdn-avatars.huggingface.co/v1/production/uploads/649cb9456209253a76854798/OY2_nPMMxkds2ME2j9xte.jpeg
Hao-Wen Dong
user
salu133445
66a9ae6f35070a5a52a241a3
Zhouhang Xie
admin_assigned
2024-07-31T08:27:04.736Z
6187199bb1085ab638324e82
/avatars/02281e4e5dc260fff62de34589f42b7d.svg
zhouhang xie
user
zhohanx
66a9ae6f35070a5a52a241a4
Carol Chen
66a9ae6f35070a5a52a241a5
Julian McAuley
Futga: Towards Fine-grained Music Understanding through
  Temporally-enhanced Generative Augmentation
Existing music captioning methods are limited to generating concise global
descriptions of short music clips, which fail to capture fine-grained musical
characteristics and time-aware musical changes. To address these limitations,
we propose FUTGA, a model equipped with fined-grained music understanding
capabilities through learning from generative augmentation with temporal
compositions. We leverage existing music caption datasets and large language
models (LLMs) to synthesize fine-grained music captions with structural
descriptions and time boundaries for full-length songs. Augmented by the
proposed synthetic dataset, FUTGA is enabled to identify the music's temporal
changes at key transition points and their musical functions, as well as
generate detailed descriptions for each music segment. We further introduce a
full-length music caption dataset generated by FUTGA, as the augmentation of
the MusicCaps and the Song Describer datasets. We evaluate the automatically
generated captions on several downstream tasks, including music generation and
retrieval. The experiments demonstrate the quality of the generated captions
and the better performance in various downstream tasks achieved by the proposed
music captioning approach. Our code and datasets can be found in
https://huggingface.co/JoshuaW1997/FUTGA{blue{https://huggingface.co/JoshuaW1997/FUTGA}}.
FUTGA enhances music captioning by generating detailed, temporally-aware descriptions using generative augmentation with temporal compositions and large language models.
fined-grained music understanding generative augmentation temporal compositions full-length music captions structural descriptions time boundaries key transition points musical functions music caption dataset MusicCaps Song Describer music generation music retrieval