AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large
  Language Models
The rapid advancement and expanding applications of Audio Large Language
Models (ALLMs) demand a rigorous understanding of their trustworthiness.
However, systematic research on evaluating these models, particularly
concerning risks unique to the audio modality, remains largely unexplored.
Existing evaluation frameworks primarily focus on the text modality or address
only a restricted set of safety dimensions, failing to adequately account for
the unique characteristics and application scenarios inherent to the audio
modality. We introduce AudioTrust-the first multifaceted trustworthiness
evaluation framework and benchmark specifically designed for ALLMs. AudioTrust
facilitates assessments across six key dimensions: fairness, hallucination,
safety, privacy, robustness, and authentication. To comprehensively evaluate
these dimensions, AudioTrust is structured around 18 distinct experimental
setups. Its core is a meticulously constructed dataset of over 4,420 audio/text
samples, drawn from real-world scenarios (e.g., daily conversations, emergency
calls, voice assistant interactions), specifically designed to probe the
multifaceted trustworthiness of ALLMs. For assessment, the benchmark carefully
designs 9 audio-specific evaluation metrics, and we employ a large-scale
automated pipeline for objective and scalable scoring of model outputs.
Experimental results reveal the trustworthiness boundaries and limitations of
current state-of-the-art open-source and closed-source ALLMs when confronted
with various high-risk audio scenarios, offering valuable insights for the
secure and trustworthy deployment of future audio models. Our platform and
benchmark are available at https://github.com/JusperLee/AudioTrust.
6833d9cfdf7cbb5c087cb9cd
Kai Li
claimed_verified
2025-08-20T08:53:57.554Z
6387676c23da90491eb9fb16
https://cdn-avatars.huggingface.co/v1/production/uploads/1669818175965-noauth.jpeg
Kai Li
user
JusperLee
6833d9cfdf7cbb5c087cb9ce
Can Shen
6833d9cfdf7cbb5c087cb9cf
Yile Liu
6833d9cfdf7cbb5c087cb9d0
Jirui Han
6833d9cfdf7cbb5c087cb9d1
Kelong Zheng
6833d9cfdf7cbb5c087cb9d2
Xuechao Zou
6833d9cfdf7cbb5c087cb9d3
Zhe Wang
6833d9cfdf7cbb5c087cb9d4
Xingjian Du
6833d9cfdf7cbb5c087cb9d5
Shun Zhang
6833d9cfdf7cbb5c087cb9d6
Hanjun Luo
6833d9cfdf7cbb5c087cb9d7
Yingbin Jin
6833d9cfdf7cbb5c087cb9d8
Xinxin Xing
6833d9cfdf7cbb5c087cb9d9
Ziyang Ma
6833d9cfdf7cbb5c087cb9da
Yue Liu
6833d9cfdf7cbb5c087cb9db
Xiaojun Jia
claimed_verified
2025-05-26T08:09:37.847Z
64c6627d5671d42e0adfad56
/avatars/8b98054b2911b86dcc4856a15306e60f.svg
jiaxiaojunQAQ
user
jiaxiaojunQAQ
6833d9cfdf7cbb5c087cb9dc
Yifan Zhang
6833d9cfdf7cbb5c087cb9dd
Junfeng Fang
6833d9cfdf7cbb5c087cb9de
Kun Wang
6833d9cfdf7cbb5c087cb9df
Yibo Yan
6833d9cfdf7cbb5c087cb9e0
Haoyang Li
6833d9cfdf7cbb5c087cb9e1
Yiming Li
6833d9cfdf7cbb5c087cb9e2
Xiaobin Zhuang
claimed_verified
2025-06-03T08:51:08.135Z
63774ca43a63a2983ffc12f9
https://cdn-avatars.huggingface.co/v1/production/uploads/63774ca43a63a2983ffc12f9/jUJasGf5_2rwiEun_eYl7.png
xiaobin zhuang
user
xiaobinzhuang
6833d9cfdf7cbb5c087cb9e3
Yang Liu
6833d9cfdf7cbb5c087cb9e4
Haibo Hu
6833d9cfdf7cbb5c087cb9e5
Zhuo Chen
6833d9cfdf7cbb5c087cb9e6
Zhizheng Wu
6833d9cfdf7cbb5c087cb9e7
Xiaolin Hu
6833d9cfdf7cbb5c087cb9e8
Eng-Siong Chng
6833d9cfdf7cbb5c087cb9e9
XiaoFeng Wang
6833d9cfdf7cbb5c087cb9ea
Wenyuan Xu
6833d9cfdf7cbb5c087cb9eb
Wei Dong
6833d9cfdf7cbb5c087cb9ec
Xinfeng Li
AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large
  Language Models
The rapid advancement and expanding applications of Audio Large Language
Models (ALLMs) demand a rigorous understanding of their trustworthiness.
However, systematic research on evaluating these models, particularly
concerning risks unique to the audio modality, remains largely unexplored.
Existing evaluation frameworks primarily focus on the text modality or address
only a restricted set of safety dimensions, failing to adequately account for
the unique characteristics and application scenarios inherent to the audio
modality. We introduce AudioTrust-the first multifaceted trustworthiness
evaluation framework and benchmark specifically designed for ALLMs. AudioTrust
facilitates assessments across six key dimensions: fairness, hallucination,
safety, privacy, robustness, and authentication. To comprehensively evaluate
these dimensions, AudioTrust is structured around 18 distinct experimental
setups. Its core is a meticulously constructed dataset of over 4,420 audio/text
samples, drawn from real-world scenarios (e.g., daily conversations, emergency
calls, voice assistant interactions), specifically designed to probe the
multifaceted trustworthiness of ALLMs. For assessment, the benchmark carefully
designs 9 audio-specific evaluation metrics, and we employ a large-scale
automated pipeline for objective and scalable scoring of model outputs.
Experimental results reveal the trustworthiness boundaries and limitations of
current state-of-the-art open-source and closed-source ALLMs when confronted
with various high-risk audio scenarios, offering valuable insights for the
secure and trustworthy deployment of future audio models. Our platform and
benchmark are available at https://github.com/JusperLee/AudioTrust.
AudioTrust evaluates the trustworthiness of Audio Large Language Models across multifaceted dimensions, using a comprehensive dataset and specific metrics to assess their performance in real-world audio scenarios.
Audio Large Language Models ALLMs trustworthiness fairness hallucination safety privacy robustness authentication AudioTrust experimental setups audio-specific evaluation metrics automated pipeline