CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction
  Following
Recent advances in audio-text large language models (LLMs) have opened new
possibilities for music understanding and generation. However, existing
benchmarks are limited in scope, often relying on simplified tasks or
multi-choice evaluations that fail to reflect the complexity of real-world
music analysis. We reinterpret a broad range of traditional MIR annotations as
instruction-following formats and introduce CMI-Bench, a comprehensive music
instruction following benchmark designed to evaluate audio-text LLMs on a
diverse set of music information retrieval (MIR) tasks. These include genre
classification, emotion regression, emotion tagging, instrument classification,
pitch estimation, key detection, lyrics transcription, melody extraction, vocal
technique recognition, instrument performance technique detection, music
tagging, music captioning, and (down)beat tracking: reflecting core challenges
in MIR research. Unlike previous benchmarks, CMI-Bench adopts standardized
evaluation metrics consistent with previous state-of-the-art MIR models,
ensuring direct comparability with supervised approaches. We provide an
evaluation toolkit supporting all open-source audio-textual LLMs, including
LTU, Qwen-audio, SALMONN, MusiLingo, etc. Experiment results reveal significant
performance gaps between LLMs and supervised models, along with their culture,
chronological and gender bias, highlighting the potential and limitations of
current models in addressing MIR tasks. CMI-Bench establishes a unified
foundation for evaluating music instruction following, driving progress in
music-aware LLMs.
6852b1597eb90a35d35de603
Yinghao Ma
6852b1597eb90a35d35de604
Siyou Li
claimed_verified
2025-07-03T07:41:52.418Z
62d53e556dff888212d58724
https://cdn-avatars.huggingface.co/v1/production/uploads/62d53e556dff888212d58724/kJxZGBvR0iMnmGC890g-e.png
Siyou Li
user
SiyouLi
6852b1597eb90a35d35de605
Juntao Yu
6852b1597eb90a35d35de606
Emmanouil Benetos
6852b1597eb90a35d35de607
Akira Maezawa
CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction
  Following
Recent advances in audio-text large language models (LLMs) have opened new
possibilities for music understanding and generation. However, existing
benchmarks are limited in scope, often relying on simplified tasks or
multi-choice evaluations that fail to reflect the complexity of real-world
music analysis. We reinterpret a broad range of traditional MIR annotations as
instruction-following formats and introduce CMI-Bench, a comprehensive music
instruction following benchmark designed to evaluate audio-text LLMs on a
diverse set of music information retrieval (MIR) tasks. These include genre
classification, emotion regression, emotion tagging, instrument classification,
pitch estimation, key detection, lyrics transcription, melody extraction, vocal
technique recognition, instrument performance technique detection, music
tagging, music captioning, and (down)beat tracking: reflecting core challenges
in MIR research. Unlike previous benchmarks, CMI-Bench adopts standardized
evaluation metrics consistent with previous state-of-the-art MIR models,
ensuring direct comparability with supervised approaches. We provide an
evaluation toolkit supporting all open-source audio-textual LLMs, including
LTU, Qwen-audio, SALMONN, MusiLingo, etc. Experiment results reveal significant
performance gaps between LLMs and supervised models, along with their culture,
chronological and gender bias, highlighting the potential and limitations of
current models in addressing MIR tasks. CMI-Bench establishes a unified
foundation for evaluating music instruction following, driving progress in
music-aware LLMs.
CMI-Bench introduces a comprehensive instruction-following benchmark for audio-text LLMs to evaluate them on a diverse range of music information retrieval tasks.
audio-text large language models LLMs music information retrieval MIR genre classification emotion regression instrument classification pitch estimation key detection lyrics transcription melody extraction vocal technique recognition instrument performance technique detection music tagging music captioning beat tracking evaluation metrics cultural bias chronological bias gender bias