Towards Holistic Evaluation of Large Audio-Language Models: A
  Comprehensive Survey
With advancements in large audio-language models (LALMs), which enhance large
language models (LLMs) with auditory capabilities, these models are expected to
demonstrate universal proficiency across various auditory tasks. While numerous
benchmarks have emerged to assess LALMs' performance, they remain fragmented
and lack a structured taxonomy. To bridge this gap, we conduct a comprehensive
survey and propose a systematic taxonomy for LALM evaluations, categorizing
them into four dimensions based on their objectives: (1) General Auditory
Awareness and Processing, (2) Knowledge and Reasoning, (3) Dialogue-oriented
Ability, and (4) Fairness, Safety, and Trustworthiness. We provide detailed
overviews within each category and highlight challenges in this field, offering
insights into promising future directions. To the best of our knowledge, this
is the first survey specifically focused on the evaluations of LALMs, providing
clear guidelines for the community. We will release the collection of the
surveyed papers and actively maintain it to support ongoing advancements in the
field.
6830745670e219f5de8ad360
Chih-Kai Yang
extracted_confirmed
2025-05-23T13:21:28.397Z
646fa3016441111304fec68d
/avatars/923629340f3785ae8c6e52cf3674d5c2.svg
Chih-Kai Yang
user
zenyn
6830745670e219f5de8ad361
Neo S. Ho
6830745670e219f5de8ad362
Hung-yi Lee
admin_assigned
2025-05-27T12:40:58.251Z
629b6eb3ad498ab19282ae6f
/avatars/adade8c512122120b06e5c50724a178d.svg
Hung-yi Lee
user
hungyilee
Towards Holistic Evaluation of Large Audio-Language Models: A
  Comprehensive Survey
With advancements in large audio-language models (LALMs), which enhance large
language models (LLMs) with auditory capabilities, these models are expected to
demonstrate universal proficiency across various auditory tasks. While numerous
benchmarks have emerged to assess LALMs' performance, they remain fragmented
and lack a structured taxonomy. To bridge this gap, we conduct a comprehensive
survey and propose a systematic taxonomy for LALM evaluations, categorizing
them into four dimensions based on their objectives: (1) General Auditory
Awareness and Processing, (2) Knowledge and Reasoning, (3) Dialogue-oriented
Ability, and (4) Fairness, Safety, and Trustworthiness. We provide detailed
overviews within each category and highlight challenges in this field, offering
insights into promising future directions. To the best of our knowledge, this
is the first survey specifically focused on the evaluations of LALMs, providing
clear guidelines for the community. We will release the collection of the
surveyed papers and actively maintain it to support ongoing advancements in the
field.
A survey proposes a systematic taxonomy for evaluating large audio-language models across dimensions including auditory awareness, knowledge reasoning, dialogue ability, and fairness, to address fragmented benchmarks in the field.
large audio-language models LALMs large language models LLMs auditory capabilities general auditory awareness processing knowledge and reasoning dialogue-oriented ability fairness safety trustworthiness taxonomy evaluations benchmark survey guidelines