AHELM: A Holistic Evaluation of Audio-Language Models
Evaluations of audio-language models (ALMs) -- multimodal models that take
interleaved audio and text as input and output text -- are hindered by the lack
of standardized benchmarks; most benchmarks measure only one or two
capabilities and omit evaluative aspects such as fairness or safety.
Furthermore, comparison across models is difficult as separate evaluations test
a limited number of models and use different prompting methods and inference
parameters. To address these shortfalls, we introduce AHELM, a benchmark that
aggregates various datasets -- including 2 new synthetic audio-text datasets
called PARADE, which evaluates the ALMs on avoiding stereotypes, and
CoRe-Bench, which measures reasoning over conversational audio through
inferential multi-turn question answering -- to holistically measure the
performance of ALMs across 10 aspects we have identified as important to the
development and usage of ALMs: audio perception, knowledge, reasoning, emotion
detection, bias, fairness, multilinguality, robustness, toxicity, and safety.
We also standardize the prompts, inference parameters, and evaluation metrics
to ensure equitable comparisons across models. We test 14 open-weight and
closed-API ALMs from 3 developers and 3 additional simple baseline systems each
consisting of an automatic speech recognizer and a language model. Our results
show that while Gemini 2.5 Pro ranks top in 5 out of 10 aspects, it exhibits
group unfairness (p=0.01) on ASR tasks whereas most of the other models do
not. We also find that the baseline systems perform reasonably well on AHELM,
with one ranking 5th overall despite having only speech-to-text capabilities.
For transparency, all raw prompts, model generations, and outputs are available
on our website at https://crfm.stanford.edu/helm/audio/v1.0.0. AHELM is
intended to be a living benchmark and new datasets and models will be added
over time.
68b51220851c6e7b001eca69
Tony Lee
68b51220851c6e7b001eca6a
Haoqin Tu
claimed_verified
2025-09-02T07:57:03.817Z
604ae011caabafacfa48e3de
https://cdn-avatars.huggingface.co/v1/production/uploads/1615519738679-noauth.jpeg
Haoqin Tu
user
PahaII
68b51220851c6e7b001eca6b
Chi Heem Wong
68b51220851c6e7b001eca6c
Zijun Wang
68b51220851c6e7b001eca6d
Siwei Yang
68b51220851c6e7b001eca6e
Yifan Mai
68b51220851c6e7b001eca6f
Yuyin Zhou
68b51220851c6e7b001eca70
Cihang Xie
claimed_verified
2025-09-03T08:29:26.054Z
645eb61da3c5cd8a16efffff
/avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
Cihang Xie
user
cihangxie
68b51220851c6e7b001eca71
Percy Liang
AHELM: A Holistic Evaluation of Audio-Language Models
Evaluations of audio-language models (ALMs) -- multimodal models that take
interleaved audio and text as input and output text -- are hindered by the lack
of standardized benchmarks; most benchmarks measure only one or two
capabilities and omit evaluative aspects such as fairness or safety.
Furthermore, comparison across models is difficult as separate evaluations test
a limited number of models and use different prompting methods and inference
parameters. To address these shortfalls, we introduce AHELM, a benchmark that
aggregates various datasets -- including 2 new synthetic audio-text datasets
called PARADE, which evaluates the ALMs on avoiding stereotypes, and
CoRe-Bench, which measures reasoning over conversational audio through
inferential multi-turn question answering -- to holistically measure the
performance of ALMs across 10 aspects we have identified as important to the
development and usage of ALMs: audio perception, knowledge, reasoning, emotion
detection, bias, fairness, multilinguality, robustness, toxicity, and safety.
We also standardize the prompts, inference parameters, and evaluation metrics
to ensure equitable comparisons across models. We test 14 open-weight and
closed-API ALMs from 3 developers and 3 additional simple baseline systems each
consisting of an automatic speech recognizer and a language model. Our results
show that while Gemini 2.5 Pro ranks top in 5 out of 10 aspects, it exhibits
group unfairness (p=0.01) on ASR tasks whereas most of the other models do
not. We also find that the baseline systems perform reasonably well on AHELM,
with one ranking 5th overall despite having only speech-to-text capabilities.
For transparency, all raw prompts, model generations, and outputs are available
on our website at https://crfm.stanford.edu/helm/audio/v1.0.0. AHELM is
intended to be a living benchmark and new datasets and models will be added
over time.
AHELM is a comprehensive benchmark for audio-language models that evaluates multiple aspects including fairness, safety, and reasoning across various datasets and models.
audio-language models multimodal models standardized benchmarks audio perception knowledge reasoning emotion detection bias fairness multilinguality robustness toxicity safety AHELM PARADE CoRe-Bench automatic speech recognizer language model Gemini 2.5 Pro group unfairness