[Paper Review] Qwen2-Audio Technical Report

Yunfei Chu, Jin Xu+10 more|arXiv (Cornell University)|Jul 15, 2024

Music Technology and Sound Studies4 citations

TL;DR

Qwen2-Audio is a large audio-language model that achieves state-of-the-art performance in audio understanding and voice interaction by leveraging natural language prompts in pre-training, instruction tuning, and direct preference optimization. It excels in both voice chat and audio analysis modes without task-specific fine-tuning, outperforming prior SOTA models like Gemini-1.5-pro on the AIR-Bench chat benchmark and achieving SOTA results on ASR, speech translation, and audio classification tasks.

ABSTRACT

We introduce the latest progress of Qwen-Audio, a large-scale audio-language model called Qwen2-Audio, which is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. In contrast to complex hierarchical tags, we have simplified the pre-training process by utilizing natural language prompts for different data and tasks, and have further expanded the data volume. We have boosted the instruction-following capability of Qwen2-Audio and implemented two distinct audio interaction modes for voice chat and audio analysis. In the voice chat mode, users can freely engage in voice interactions with Qwen2-Audio without text input. In the audio analysis mode, users could provide audio and text instructions for analysis during the interaction. Note that we do not use any system prompts to switch between voice chat and audio analysis modes. Qwen2-Audio is capable of intelligently comprehending the content within audio and following voice commands to respond appropriately. For instance, in an audio segment that simultaneously contains sounds, multi-speaker conversations, and a voice command, Qwen2-Audio can directly understand the command and provide an interpretation and response to the audio. Additionally, DPO has optimized the model's performance in terms of factuality and adherence to desired behavior. According to the evaluation results from AIR-Bench, Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests focused on audio-centric instruction-following capabilities. Qwen2-Audio is open-sourced with the aim of fostering the advancement of the multi-modal language community.

Motivation & Objective

• To enhance the instruction-following capability of large audio-language models (LALMs) for diverse audio inputs.
• To enable seamless, mode-agnostic interaction through two unified audio interaction modes: voice chat and audio analysis.
• To simplify the pre-training pipeline by replacing hierarchical task tags with natural language prompts, improving training efficiency and generalization.
• To improve response quality and factual consistency through direct preference optimization (DPO).
• To achieve SOTA performance across multiple audio-centric benchmarks without task-specific fine-tuning.

Proposed method

• Pre-training uses natural language prompts to unify diverse data and tasks, replacing complex hierarchical tagging.
• The audio encoder is initialized from Whisper-large-v3 and processes 16kHz audio into 128-channel mel-spectrograms with 40ms temporal resolution.
• A two-stage training process includes supervised fine-tuning (SFT) with high-quality, diverse instruction data and direct preference optimization (DPO) to align outputs with human preferences.
• The model employs a conditional language modeling objective: maximizing the probability of next text tokens given audio representations and prior text tokens.
• Audio input is processed via a pooling layer with stride 2 to reduce sequence length while preserving temporal resolution.
• The model operates in two functional modes—voice chat and audio analysis—without requiring explicit system prompts to switch.

Figure 1 : Performance of Qwen2-Audio, Qwen-Audio and previous top-tiers from LALMs such as SpeechT5 (Ao et al., 2021 ) , SpeechNet (Chen et al., 2021 ) , SpeechLLaMA (Wu et al., 2023a ) , SALMONN (Tang et al., 2024 ) , Whisper (Radford et al., 2023 ) Pengi (Deshmukh et al., 2023 ) , and SpeechVerse

Experimental results

Research questions

• RQ1 Can natural language prompts replace hierarchical task tags in pre-training to simplify the training pipeline and improve generalization in LALMs?
• RQ2 To what extent can a unified audio-language model achieve strong performance in both voice chat and audio analysis without task-specific fine-tuning?
• RQ3 How does direct preference optimization (DPO) improve factual consistency and alignment with human preferences in audio-language generation?
• RQ4 What is the performance of Qwen2-Audio on multi-domain audio benchmarks, particularly in complex, mixed-audio scenarios?
• RQ5 Does Qwen2-Audio outperform existing SOTA models like Gemini-1.5-pro and Whisper-large-v3 in instruction-following and audio understanding tasks?

Key findings

• Qwen2-Audio achieves a 1.6% WER on LibriSpeech test-clean and 3.6% WER on test-other, outperforming previous models.
• On CoVoST2, Qwen2-Audio achieves the highest average BLEU score across all seven translation directions, surpassing prior baselines.
• On the VSC task, Qwen2-Audio achieves 0.9392 accuracy, significantly outperforming Qwen-Audio (0.9289) and Pengi (0.6035).
• On the AIR-Bench chat benchmark, Qwen2-Audio scores 7.18 in the speech subset, 6.99 in sound, 6.79 in music, and 6.77 in mixed-audio—surpassing Gemini-1.5-pro and other SOTA models.
• In the SER task, Qwen2-Audio achieves 0.553 accuracy on Meld, outperforming Qwen-Audio (0.557) and surpassing WavLM-large (0.542).
• Qwen2-Audio demonstrates robustness in mixed-audio analysis, correctly identifying commands and sounds in complex, multi-source audio clips without explicit segmentation.

Figure 2 : The overview of three-stage training process of Qwen2-Audio.