[Paper Review] Audiobox: Unified Audio Generation with Natural Language Prompts

Apoorv Vyas, Bowen Shi+22 more|arXiv (Cornell University)|Dec 25, 2023

Music and Audio Processing5 citations

TL;DR

Audiobox is a unified audio generation model based on flow-matching that enables controllable, high-quality synthesis of speech, music, and sound effects using natural language prompts. It achieves state-of-the-art results in zero-shot text-to-speech (0.745 Librispeech similarity) and text-to-sound generation (0.77 FAD on AudioCaps), with improved controllability through description-based and example-based prompting, and enhanced inference speed via Bespoke Solvers.

ABSTRACT

Audio is an essential part of our life, but creating it often requires expertise and is time-consuming. Research communities have made great progress over the past year advancing the performance of large scale audio generative models for a single modality (speech, sound, or music) through adopting more powerful generative models and scaling data. However, these models lack controllability in several aspects: speech generation models cannot synthesize novel styles based on text description and are limited on domain coverage such as outdoor environments; sound generation models only provide coarse-grained control based on descriptions like "a person speaking" and would only generate mumbling human voices. This paper presents Audiobox, a unified model based on flow-matching that is capable of generating various audio modalities. We design description-based and example-based prompting to enhance controllability and unify speech and sound generation paradigms. We allow transcript, vocal, and other audio styles to be controlled independently when generating speech. To improve model generalization with limited labels, we adapt a self-supervised infilling objective to pre-train on large quantities of unlabeled audio. Audiobox sets new benchmarks on speech and sound generation (0.745 similarity on Librispeech for zero-shot TTS; 0.77 FAD on AudioCaps for text-to-sound) and unlocks new methods for generating audio with novel vocal and acoustic styles. We further integrate Bespoke Solvers, which speeds up generation by over 25 times compared to the default ODE solver for flow-matching, without loss of performance on several tasks. Our demo is available at https://audiobox.metademolab.com/

Motivation & Objective

• To address the lack of controllability and modality-specific limitations in existing audio generative models.
• To unify speech and sound generation under a single framework with both description-based and example-based prompting.
• To improve generalization on limited-labeled data using a self-supervised infilling objective.
• To enable fine-grained control over vocal styles, transcripts, and audio attributes in speech generation.
• To develop a scalable, general-purpose audio generation model that supports diverse, in-the-wild audio content.

Proposed method

• A flow-matching-based diffusion model architecture enables end-to-end audio generation from text or audio prompts.
• Description-based prompting allows control over textual content, vocal style, and acoustic environment independently.
• Example-based prompting enables style transfer using short audio clips, enhancing style fidelity.
• A self-supervised infilling objective is adapted for pre-training on large-scale unlabeled audio to improve generalization.
• Bespoke Solvers are introduced to accelerate inference by over 25x compared to standard ODE solvers without performance loss.
• Joint-CLAP is used to align audio and text embeddings, improving zero-shot generalization across modalities.

Experimental results

Research questions

• RQ1 Can a unified audio generation model effectively handle diverse modalities—speech, music, and sound—using natural language prompts?
• RQ2 How can both description-based and example-based prompting be integrated to improve controllability in speech generation?
• RQ3 To what extent can self-supervised infilling pre-training improve performance on low-resource audio generation tasks?
• RQ4 What is the impact of specialized inference solvers like Bespoke Solvers on generation speed and quality?
• RQ5 Can a single model achieve state-of-the-art performance across zero-shot text-to-speech and text-to-sound benchmarks?

Key findings

• Audiobox achieves a Librispeech similarity score of 0.745 on zero-shot text-to-speech generation, setting a new SOTA benchmark.
• On the AudioCaps text-to-sound generation benchmark, Audiobox achieves a Fréchet Audio Distance (FAD) of 0.77, outperforming prior methods.
• The model enables independent control over transcript, vocal style, and acoustic environment in speech generation, enabling novel vocal and acoustic styles.
• Bespoke Solvers accelerate audio generation by over 25 times compared to standard ODE solvers while maintaining generation quality.
• The self-supervised infilling objective significantly improves generalization on limited-labeled data, especially for rare or complex audio patterns.
• Joint-CLAP alignment improves zero-shot transfer performance across speech, music, and sound generation tasks.