MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
The ability to comprehend audio--which includes speech, non-speech sounds,
and music--is crucial for AI agents to interact effectively with the world. We
present MMAU, a novel benchmark designed to evaluate multimodal audio
understanding models on tasks requiring expert-level knowledge and complex
reasoning. MMAU comprises 10k carefully curated audio clips paired with
human-annotated natural language questions and answers spanning speech,
environmental sounds, and music. It includes information extraction and
reasoning questions, requiring models to demonstrate 27 distinct skills across
unique and challenging tasks. Unlike existing benchmarks, MMAU emphasizes
advanced perception and reasoning with domain-specific knowledge, challenging
models to tackle tasks akin to those faced by experts. We assess 18 open-source
and proprietary (Large) Audio-Language Models, demonstrating the significant
challenges posed by MMAU. Notably, even the most advanced Gemini Pro v1.5
achieves only 52.97% accuracy, and the state-of-the-art open-source Qwen2-Audio
achieves only 52.50%, highlighting considerable room for improvement. We
believe MMAU will drive the audio and multimodal research community to develop
more advanced audio understanding models capable of solving complex audio
tasks.
671ef25002737c628d38c1a3
S Sakshi
claimed_verified
2024-10-30T09:54:36.298Z
66607692fc751a711a6efbd9
/avatars/f04f59822d5eaf64ccb205186aef3a50.svg
Sakshi
user
ssakshi
671ef25002737c628d38c1a4
Utkarsh Tyagi
671ef25002737c628d38c1a5
Sonal Kumar
claimed_verified
2024-10-28T15:40:32.700Z
65203f31d6dec04046139874
https://cdn-avatars.huggingface.co/v1/production/uploads/65203f31d6dec04046139874/1LDWW4KblYeIDve9F_kUA.png
Sonal Kumar
user
sonalkum
671ef25002737c628d38c1a6
Ashish Seth
671ef25002737c628d38c1a7
Ramaneswaran Selvakumar
671ef25002737c628d38c1a8
Oriol Nieto
admin_assigned
2024-10-28T15:28:06.888Z
657a29d7d3e458405b8dc146
/avatars/1fe425dcf3133462fb6376e8e1550b4a.svg
Oriol Nieto
user
urinieto
671ef25002737c628d38c1a9
Ramani Duraiswami
admin_assigned
2024-10-28T15:28:01.454Z
65b8f0339b7250e205100190
/avatars/29df91154c5911a44ef503bb3580ac81.svg
Ramani Duraiswami
user
RamaniD
671ef25002737c628d38c1aa
Sreyan Ghosh
claimed_verified
2024-10-28T15:36:42.210Z
62c9664eb34e600d7eaa4beb
/avatars/ca23ecdec2d31c99ecce97d9b180ae0c.svg
Ghosh
user
Sreyan88
671ef25002737c628d38c1ab
Dinesh Manocha
admin_assigned
2024-10-28T15:27:27.197Z
6537a569568d8be8fa096b8c
/avatars/bfda5cb252d8b5bc3ad737d99c0d7f49.svg
Dinesh Manocha
user
manocha
MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
The ability to comprehend audio--which includes speech, non-speech sounds,
and music--is crucial for AI agents to interact effectively with the world. We
present MMAU, a novel benchmark designed to evaluate multimodal audio
understanding models on tasks requiring expert-level knowledge and complex
reasoning. MMAU comprises 10k carefully curated audio clips paired with
human-annotated natural language questions and answers spanning speech,
environmental sounds, and music. It includes information extraction and
reasoning questions, requiring models to demonstrate 27 distinct skills across
unique and challenging tasks. Unlike existing benchmarks, MMAU emphasizes
advanced perception and reasoning with domain-specific knowledge, challenging
models to tackle tasks akin to those faced by experts. We assess 18 open-source
and proprietary (Large) Audio-Language Models, demonstrating the significant
challenges posed by MMAU. Notably, even the most advanced Gemini Pro v1.5
achieves only 52.97% accuracy, and the state-of-the-art open-source Qwen2-Audio
achieves only 52.50%, highlighting considerable room for improvement. We
believe MMAU will drive the audio and multimodal research community to develop
more advanced audio understanding models capable of solving complex audio
tasks.
MMAU, a new benchmark for evaluating advanced audio understanding models, comprises audio clips with complex reasoning tasks and requires models to demonstrate specialized skills, highlighting significant areas for improvement in current models.
multimodal audio understanding information extraction audio-language models domain-specific knowledge audio clips natural language questions advanced perception reasoning tasks