AU-Harness: An Open-Source Toolkit for Holistic Evaluation of Audio LLMs
Large Audio Language Models (LALMs) are rapidly advancing, but evaluating
them remains challenging due to inefficient toolkits that limit fair comparison
and systematic assessment. Current frameworks suffer from three critical
issues: slow processing that bottlenecks large-scale studies, inconsistent
prompting that hurts reproducibility, and narrow task coverage that misses
important audio reasoning capabilities. We introduce AU-Harness, an efficient
and comprehensive evaluation framework for LALMs. Our system achieves a speedup
of up to 127% over existing toolkits through optimized batch processing and
parallel execution, enabling large-scale evaluations previously impractical. We
provide standardized prompting protocols and flexible configurations for fair
model comparison across diverse scenarios. Additionally, we introduce two new
evaluation categories: LLM-Adaptive Diarization for temporal audio
understanding and Spoken Language Reasoning for complex audio-based cognitive
tasks. Through evaluation across 380+ tasks, we reveal significant gaps in
current LALMs, particularly in temporal understanding and complex spoken
language reasoning tasks. Our findings also highlight a lack of standardization
in instruction modality existent across audio benchmarks, which can lead up
performance differences up to 9.5 absolute points on the challenging complex
instruction following downstream tasks. AU-Harness provides both practical
evaluation tools and insights into model limitations, advancing systematic LALM
development.
68c4535eee0eed1697d6b0b8
Sidharth Surapaneni
68c4535eee0eed1697d6b0b9
Hoang Nguyen
68c4535eee0eed1697d6b0ba
Jash Mehta
68c4535eee0eed1697d6b0bb
Aman Tiwari
claimed_verified
2025-09-13T14:56:12.143Z
62d913739a5353eef9d7edf3
https://cdn-avatars.huggingface.co/v1/production/uploads/62d913739a5353eef9d7edf3/pRgen2izGJle3ahupOdC7.jpeg
Aman Tiwari
user
amant555
68c4535eee0eed1697d6b0bc
Oluwanifemi Bamgbose
68c4535eee0eed1697d6b0bd
Akshay Kalkunte
68c4535eee0eed1697d6b0be
Sai Rajeswar
68c4535eee0eed1697d6b0bf
Sathwik Tejaswi Madhusudhan
AU-Harness: An Open-Source Toolkit for Holistic Evaluation of Audio LLMs
Large Audio Language Models (LALMs) are rapidly advancing, but evaluating
them remains challenging due to inefficient toolkits that limit fair comparison
and systematic assessment. Current frameworks suffer from three critical
issues: slow processing that bottlenecks large-scale studies, inconsistent
prompting that hurts reproducibility, and narrow task coverage that misses
important audio reasoning capabilities. We introduce AU-Harness, an efficient
and comprehensive evaluation framework for LALMs. Our system achieves a speedup
of up to 127% over existing toolkits through optimized batch processing and
parallel execution, enabling large-scale evaluations previously impractical. We
provide standardized prompting protocols and flexible configurations for fair
model comparison across diverse scenarios. Additionally, we introduce two new
evaluation categories: LLM-Adaptive Diarization for temporal audio
understanding and Spoken Language Reasoning for complex audio-based cognitive
tasks. Through evaluation across 380+ tasks, we reveal significant gaps in
current LALMs, particularly in temporal understanding and complex spoken
language reasoning tasks. Our findings also highlight a lack of standardization
in instruction modality existent across audio benchmarks, which can lead up
performance differences up to 9.5 absolute points on the challenging complex
instruction following downstream tasks. AU-Harness provides both practical
evaluation tools and insights into model limitations, advancing systematic LALM
development.
AU-Harness is an efficient and comprehensive evaluation framework for Large Audio Language Models (LALMs) that addresses issues of speed, reproducibility, and task coverage, revealing gaps in temporal understanding and spoken language reasoning.
LALMs AU-Harness batch processing parallel execution standardized prompting LLM-Adaptive Diarization Spoken Language Reasoning temporal audio understanding complex audio-based cognitive tasks instruction modality