[Paper Review] AudioBench: A Universal Benchmark for Audio Large Language Models

Bin Wang, Xunlong Zou+7 more|arXiv (Cornell University)|Jun 23, 2024

Music and Audio Processing4 citations

TL;DR

AudioBench is a comprehensive benchmark for evaluating audio large language models (AudioLLMs) across 8 diverse tasks and 26 datasets, including speech understanding, voice interpretation, and audio scene analysis. It evaluates four open-source models using standardized prompts, diverse input lengths, and novel evaluation metrics—finding no model excels across all tasks, highlighting significant room for improvement in AudioLLM capabilities.

ABSTRACT

We introduce AudioBench, a universal benchmark designed to evaluate Audio Large Language Models (AudioLLMs). It encompasses 8 distinct tasks and 26 datasets, among which, 7 are newly proposed datasets. The evaluation targets three main aspects: speech understanding, audio scene understanding, and voice understanding (paralinguistic). Despite recent advancements, there lacks a comprehensive benchmark for AudioLLMs on instruction following capabilities conditioned on audio signals. AudioBench addresses this gap by setting up datasets as well as desired evaluation metrics. Besides, we also evaluated the capabilities of five popular models and found that no single model excels consistently across all tasks. We outline the research outlook for AudioLLMs and anticipate that our open-sourced evaluation toolkit, data, and leaderboard will offer a robust testbed for future model developments.

Motivation & Objective

• To address the lack of comprehensive, unified benchmarks for evaluating audio large language models (AudioLLMs) across diverse audio understanding tasks.
• To provide a standardized evaluation framework with diverse datasets, prompt templates, and input lengths to ensure fair and holistic model comparison.
• To develop reliable, cost-effective evaluation metrics for open-ended audio generation, reducing reliance on expensive or unstable LLM judges like GPT-4.
• To identify performance gaps across existing AudioLLMs by testing them on real-world, instruction-following audio tasks.
• To establish a public, open-source testbed with code, data, and a leaderboard to accelerate future research and development in AudioLLMs.

Proposed method

• The benchmark includes 8 core tasks: Speech Understanding, Audio Scene Understanding, Voice Understanding, Speech Instruction, Audio Captioning, Accent Recognition, Gender Recognition, and Emotion Recognition.
• It integrates 26 datasets—7 newly curated or adapted—to cover speech, environmental sounds, paralinguistic features, and multimodal audio-text interactions.
• Evaluation uses standardized prompt templates and varies input length from seconds to minutes to test long-context capabilities.
• The benchmark includes over 400 hours of audio and 100,000+ samples, enabling comprehensive model evaluation.
• It evaluates model performance using both automatic metrics (e.g., WER, METEOR) and human-in-the-loop evaluation via model-as-judge (e.g., LLaMA-3-8B-Instruct) to assess open-ended generation quality.
• The study compares four open-sourced models: Whisper+Llama3, SALMONN, Qwen-Audio, and WavLLM, using both end-to-end and cascaded architectures.

Experimental results

Research questions

• RQ1 How well do existing AudioLLMs generalize across diverse audio understanding tasks, including speech, voice, and environmental sound comprehension?
• RQ2 What is the performance gap between end-to-end and cascaded AudioLLM architectures across different audio input types and lengths?
• RQ3 How effective are open-source model-as-judge approaches (e.g., LLaMA-3) in evaluating AudioLLM outputs compared to GPT-4, and what is their correlation with human judgment?
• RQ4 Which datasets and evaluation metrics best reflect real-world instruction-following capabilities of AudioLLMs?
• RQ5 What are the key limitations and failure modes of current AudioLLMs when faced with complex, multi-modal audio inputs?

Key findings

• No single AudioLLM achieves superior performance across all 8 tasks, indicating significant room for improvement in generalization and robustness.
• The Whisper+Llama3 cascaded model outperforms end-to-end models on several ASR tasks, with a WER of 1.83 on LibriSpeech-Clean, underlining the value of modular design.
• LLaMA-3-8B-Instruct shows higher correlation with GPT-4 as a judge than Prometheus-2, making it a more reliable and cost-effective evaluation proxy.
• The WavCaps and AudioCaps datasets show strong performance on audio captioning, with METEOR scores of 21.36 and 22.79 respectively when using the LLaMA-3 model as judge.
• On emotion and sentiment recognition tasks (e.g., IEMOCAP-Emotion, MELD), models show moderate performance, with scores around 1.5 on the M.J.8B scale, indicating challenges in paralinguistic understanding.
• The benchmark reveals that current models struggle with long-form audio inputs, as seen in the high WER (25.88) on Tedlium3-Longform, suggesting limitations in context retention.