[Paper Review] Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities

Zhifeng Kong, Arushi Goel+4 more|arXiv (Cornell University)|Feb 2, 2024

Speech and dialogue systems4 citations

TL;DR

Audio Flamingo is a novel audio language model that achieves state-of-the-art performance in audio understanding by combining efficient cross-attention fusion of audio features with large language models, enabling in-context learning, retrieval-augmented generation, and multi-turn dialogue without fine-tuning. It outperforms prior models on diverse benchmarks with significantly fewer parameters.

ABSTRACT

Augmenting large language models (LLMs) to understand audio -- including non-speech sounds and non-verbal speech -- is critically important for diverse real-world applications of LLMs. In this paper, we propose Audio Flamingo, a novel audio language model with 1) strong audio understanding abilities, 2) the ability to quickly adapt to unseen tasks via in-context learning and retrieval, and 3) strong multi-turn dialogue abilities. We introduce a series of training techniques, architecture design, and data strategies to enhance our model with these abilities. Extensive evaluations across various audio understanding tasks confirm the efficacy of our method, setting new state-of-the-art benchmarks. Our demo website is https://audioflamingo.github.io/ and the code is open-sourced at https://github.com/NVIDIA/audio-flamingo.

Motivation & Objective

• To extend large language models (LLMs) with robust audio understanding beyond speech transcription, including non-speech and non-verbal sounds.
• To enable few-shot adaptation to unseen audio tasks via in-context learning (ICL) and retrieval-augmented generation (RAG) without fine-tuning.
• To equip the model with strong multi-turn dialogue capabilities through curated, context-rich dialogue datasets and supervised fine-tuning.
• To overcome limitations in prior models that trade off performance between close-ended and open-ended audio tasks.

Proposed method

• Introduces a sliding-window audio feature extractor based on Elizalde et al. (2023b) to preserve temporal information in variable-length audio inputs.
• Uses cross-attention mechanisms to condition the LLM on audio features, achieving linear complexity with respect to audio token count, unlike quadratic approaches from prior methods.
• Employs a two-stage training pipeline: pre-training on a heterogeneous dataset of ~5.9M audio-text pairs, followed by supervised fine-tuning (SFT) on task-specific data.
• Designs a novel cross-attention mask for interleaved in-context learning samples to improve reasoning over retrieved examples.
• Creates interleaved ICL datasets using retrieved examples and a structured prompt template to enable few-shot generalization.
• Constructs two multi-turn dialogue datasets using GPT-4 to generate contextually rich, annotated dialogues, fine-tuning Audio Flamingo for conversational performance.

Figure 1 : Audio Flamingo versus previous state-of-the-art (Deshmukh et al., 2023 ; Chu et al., 2023 ; Gong et al., 2023b , c ; Tang et al., 2023a ; Ghosh et al., 2023 ) on a number of audio understanding benchmarks. The numbers are normalized such that the maximum of all models is $100\%$ on each t

Experimental results

Research questions

• RQ1 Can a single audio language model achieve strong performance across diverse audio understanding tasks, including both close-ended and open-ended benchmarks?
• RQ2 Can the model generalize to unseen tasks via in-context learning and retrieval without task-specific fine-tuning?
• RQ3 Does the model support coherent, context-aware multi-turn dialogue with strong reasoning and memory over conversation history?
• RQ4 How does the model’s architecture and training strategy compare to prior methods in terms of parameter efficiency and performance?

Key findings

• Audio Flamingo sets new state-of-the-art results on multiple audio understanding benchmarks, outperforming prior models including those by Chu et al. (2023) and Gong et al. (2023c).
• The model achieves SOTA performance with less than one-third the number of parameters compared to prior SOTA models, demonstrating high parameter efficiency.
• In few-shot learning settings, Audio Flamingo correctly identifies music genre (e.g., pop) and instrument (e.g., flute) even when retrieved examples are noisy or incorrect.
• The dialogue version of Audio Flamingo significantly outperforms baselines in multi-turn conversations, correctly identifying overlapping bird sounds and speech, and accurately describing vocal characteristics and music genres.
• The model demonstrates robustness in disambiguating complex audio scenes, such as distinguishing between bird vocalizations, animal sounds, and human speech, and correctly identifying overlapping events.
• The use of retrieval-augmented in-context learning enables accurate reasoning even when retrieved examples are partially incorrect, showing strong generalization and noise resilience.

Figure 2 : Neural architecture of Audio Flamingo. It takes interleaved audio and text as input and outputs free-form text.