Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large Language Models

Created by
  • Haebom

Author

Kai Li, Can Shen, Yile Liu, Jirui Han, Kelong Zheng, Xuechao Zou, Zhe Wang, Shun Zhang, Xingjian Du, Hanjun Luo, Yingbin Jin, Xinxin Tianwei Zhang, Yang Liu, Haibo Hu, Zhizheng Wu, Xiaolin Hu, Eng-Siong Chng, Wenyuan Xu, XiaoFeng Wang, Wei Dong, Xinfeng Li

Outline

The reliability of audio large-scale language models (ALLMs) has not been widely studied, and existing text-centric evaluation frameworks fail to adequately address the inherent vulnerabilities arising from the acoustic characteristics of audio. We identify ALLM reliability risks where nonsense acoustic cues, such as timbre, intonation, and background noise, can manipulate model behavior. We propose AudioTrust, a comprehensive framework for systematically assessing ALLM reliability against audio-specific risks. AudioTrust encompasses six key dimensions—fairness, hallucination, safety, privacy, robustness, and authentication—and implements 26 subtasks comprised of over 4,420 audio samples collected from real-world scenarios (daily conversations, emergency calls, and voice assistant interactions). Using a human-validated automated pipeline, we conduct a comprehensive evaluation across 18 experimental configurations and evaluate 14 state-of-the-art open-source and closed-source ALLMs, revealing significant limitations when faced with a variety of high-stakes audio scenarios.

Takeaways, Limitations

Takeaways:
The AudioTrust framework provides a comprehensive approach to assessing the trustworthiness of ALLM.
Uncovering the limitations of ALLM in various high-risk audio scenarios.
Provides insights into deploying secure audio models.
Contribute to research reproducibility and advancement through open code and data sharing.
Limitations:
The paper does not explicitly mention the specific Limitations (however, the ALLM Limitations covered in the paper is presented).
👍