MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic
  Evaluation of Audio General Intelligence
Audio comprehension-including speech, non-speech sounds, and music-is
essential for achieving human-level intelligence. Consequently, AI agents must
demonstrate holistic audio understanding to qualify as generally intelligent.
However, evaluating auditory intelligence comprehensively remains challenging.
To address this gap, we introduce MMAU-Pro, the most comprehensive and
rigorously curated benchmark for assessing audio intelligence in AI systems.
MMAU-Pro contains 5,305 instances, where each instance has one or more audios
paired with human expert-generated question-answer pairs, spanning speech,
sound, music, and their combinations. Unlike existing benchmarks, MMAU-Pro
evaluates auditory intelligence across 49 unique skills and multiple complex
dimensions, including long-form audio comprehension, spatial audio reasoning,
multi-audio understanding, among others. All questions are meticulously
designed to require deliberate multi-hop reasoning, including both
multiple-choice and open-ended response formats. Importantly, audio data is
sourced directly ``from the wild" rather than from existing datasets with known
distributions. We evaluate 22 leading open-source and proprietary multimodal AI
models, revealing significant limitations: even state-of-the-art models such as
Gemini 2.5 Flash and Audio Flamingo 3 achieve only 59.2% and 51.7% accuracy,
respectively, approaching random performance in multiple categories. Our
extensive analysis highlights specific shortcomings and provides novel
insights, offering actionable perspectives for the community to enhance future
AI systems' progression toward audio general intelligence. The benchmark and
code is available at https://sonalkum.github.io/mmau-pro.
68a539856cf0bf898542ec92
Sonal Kumar
claimed_verified
2025-08-28T08:56:41.879Z
65203f31d6dec04046139874
https://cdn-avatars.huggingface.co/v1/production/uploads/65203f31d6dec04046139874/1LDWW4KblYeIDve9F_kUA.png
Sonal Kumar
user
sonalkum
68a539856cf0bf898542ec93
Šimon Sedláček
68a539856cf0bf898542ec94
Vaibhavi Lokegaonkar
68a539856cf0bf898542ec95
Fernando López
claimed_verified
2025-08-28T08:56:36.936Z
6329da8459950c1d279afea2
https://cdn-avatars.huggingface.co/v1/production/uploads/1668763140756-6329da8459950c1d279afea2.jpeg
Fernando López Gavilánez
user
ferugit
68a539856cf0bf898542ec96
Wenyi Yu
68a539856cf0bf898542ec97
Nishit Anand
claimed_verified
2025-08-28T08:56:38.738Z
6316edb01cc81c5e95eb6100
https://cdn-avatars.huggingface.co/v1/production/uploads/6316edb01cc81c5e95eb6100/r0L1kMMOUQENc1vpsbyxk.jpeg
Nishit Anand
user
nishitanand
68a539856cf0bf898542ec98
Hyeonggon Ryu
68a539856cf0bf898542ec99
Lichang Chen
68a539856cf0bf898542ec9a
Maxim Plička
68a539856cf0bf898542ec9b
Miroslav Hlaváček
68a539856cf0bf898542ec9c
William Fineas Ellingwood
68a539856cf0bf898542ec9d
Sathvik Udupa
68a539856cf0bf898542ec9e
Siyuan Hou
68a539856cf0bf898542ec9f
Allison Ferner
68a539856cf0bf898542eca0
Sara Barahona
68a539856cf0bf898542eca1
Cecilia Bolaños
68a539856cf0bf898542eca2
Satish Rahi
68a539856cf0bf898542eca3
Laura Herrera-Alarcón
68a539856cf0bf898542eca4
Satvik Dixit
68a539856cf0bf898542eca5
Siddhi Patil
68a539856cf0bf898542eca6
Soham Deshmukh
68a539856cf0bf898542eca7
Lasha Koroshinadze
68a539856cf0bf898542eca8
Yao Liu
68a539856cf0bf898542eca9
Leibny Paola Garcia Perera
68a539856cf0bf898542ecaa
Eleni Zanou
68a539856cf0bf898542ecab
Themos Stafylakis
68a539856cf0bf898542ecac
Joon Son Chung
68a539856cf0bf898542ecad
David Harwath
68a539856cf0bf898542ecae
Chao Zhang
68a539856cf0bf898542ecaf
Dinesh Manocha
68a539856cf0bf898542ecb0
Alicia Lozano-Diez
68a539856cf0bf898542ecb1
Santosh Kesiraju
68a539856cf0bf898542ecb2
Sreyan Ghosh
claimed_verified
2025-08-20T08:51:15.060Z
62c9664eb34e600d7eaa4beb
/avatars/ca23ecdec2d31c99ecce97d9b180ae0c.svg
Ghosh
user
Sreyan88
68a539856cf0bf898542ecb3
Ramani Duraiswami
claimed_verified
2025-09-01T07:54:34.436Z
65b8f0339b7250e205100190
/avatars/29df91154c5911a44ef503bb3580ac81.svg
Ramani Duraiswami
user
RamaniD
MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic
  Evaluation of Audio General Intelligence
Audio comprehension-including speech, non-speech sounds, and music-is
essential for achieving human-level intelligence. Consequently, AI agents must
demonstrate holistic audio understanding to qualify as generally intelligent.
However, evaluating auditory intelligence comprehensively remains challenging.
To address this gap, we introduce MMAU-Pro, the most comprehensive and
rigorously curated benchmark for assessing audio intelligence in AI systems.
MMAU-Pro contains 5,305 instances, where each instance has one or more audios
paired with human expert-generated question-answer pairs, spanning speech,
sound, music, and their combinations. Unlike existing benchmarks, MMAU-Pro
evaluates auditory intelligence across 49 unique skills and multiple complex
dimensions, including long-form audio comprehension, spatial audio reasoning,
multi-audio understanding, among others. All questions are meticulously
designed to require deliberate multi-hop reasoning, including both
multiple-choice and open-ended response formats. Importantly, audio data is
sourced directly ``from the wild" rather than from existing datasets with known
distributions. We evaluate 22 leading open-source and proprietary multimodal AI
models, revealing significant limitations: even state-of-the-art models such as
Gemini 2.5 Flash and Audio Flamingo 3 achieve only 59.2% and 51.7% accuracy,
respectively, approaching random performance in multiple categories. Our
extensive analysis highlights specific shortcomings and provides novel
insights, offering actionable perspectives for the community to enhance future
AI systems' progression toward audio general intelligence. The benchmark and
code is available at https://sonalkum.github.io/mmau-pro.
MMAU-Pro is a comprehensive benchmark for evaluating audio intelligence in AI systems, assessing 49 unique skills across speech, sound, music, and their combinations, revealing significant limitations in current models.
MMAU-Pro long-form audio comprehension spatial audio reasoning multi-audio understanding multi-hop reasoning multimodal AI models Gemini 2.5 Flash Audio Flamingo 3