[Paper Review] Towards audio language modeling -- an overview

Haibin Wu, Xuanjun Chen+5 more|arXiv (Cornell University)|Feb 20, 2024

Speech Recognition and Synthesis4 citations

TL;DR

This paper provides a comprehensive, systematic review of neural audio codecs and codec-based audio language models (LMs), analyzing their architectures, training methods, and applications. It compares six open-source codecs and eleven codec-based LMs, highlighting their use of discrete tokenization for multimodal audio generation, speech synthesis, and zero-shot transfer, with key insights into model design, bit rate efficiency, and performance across speech, music, and general audio tasks.

ABSTRACT

Neural audio codecs are initially introduced to compress audio data into compact codes to reduce transmission latency. Researchers recently discovered the potential of codecs as suitable tokenizers for converting continuous audio into discrete codes, which can be employed to develop audio language models (LMs). Numerous high-performance neural audio codecs and codec-based LMs have been developed. The paper aims to provide a thorough and systematic overview of the neural audio codec models and codec-based LMs.

Motivation & Objective

• To address the lack of a systematic review comparing neural audio codecs and codec-based language models in the rapidly evolving field of audio representation learning.
• To analyze the training methodologies, implementation settings, and data used in six representative open-source neural codec models.
• To examine how eleven diverse codec-based language models utilize discrete audio tokens for multimodal tasks such as text-to-speech, speech-to-speech translation, and music generation.
• To provide actionable insights and research directions for future development of universal audio language models and efficient, high-fidelity neural codecs.

Proposed method

• Conducts an in-depth comparative analysis of six open-source neural codec models—SoundStream, SoundStorm, Encodec, AudioDec, AcademiCodec, and SpeechTokenizer—focusing on architecture, training objectives, and data.
• Evaluates the integration of these codecs into language models by analyzing how each codec-based LM processes discrete tokens for downstream tasks.
• Classifies and compares codec-based language models based on input modality (text, audio, phoneme, MIDI), output type, and supported tasks such as TTS, ASR, S2ST, music generation, and audio editing.
• Uses a structured framework to compare model components, including the use of residual vector quantization (RVQ), autoregressive vs. non-autoregressive decoding, and the integration of Transformer or LSTM modules for sequence modeling.
• Examines the role of pre-trained speech representations (e.g., Hubert, wav2vec 2.0) and discrete units in enabling textless speech language modeling and zero-shot transfer.
• Reviews the use of prompting techniques (e.g., in-context learning, instruction tuning) in speech LMs to extend their capabilities beyond generation to tasks like speech translation and editing.

Figure 1: Timeline of current neural codec models and codec-based language models.

Experimental results

Research questions

• RQ1 How do different neural codec architectures differ in design, training objectives, and performance across speech and audio reconstruction?
• RQ2 What are the key architectural and training differences between codecs optimized for high-fidelity audio and those designed for low bit-rate, high-efficiency applications?
• RQ3 How do codec-based language models leverage discrete audio tokens to perform diverse multimodal tasks such as text-to-speech, music generation, and speech translation?
• RQ4 What are the trade-offs between using pre-trained speech representations (e.g., Hubert) and end-to-end learned discrete units in building speech language models?
• RQ5 What are the most effective prompting and fine-tuning strategies for extending the capabilities of codec-based language models beyond generation to tasks like speech editing and translation?

Key findings

• AcademiCodec achieves a low bit rate per second (BPS) of 1.5 by using group-residual vector quantization, significantly reducing sequence length for audio language modeling.
• Encodec improves sequence modeling by integrating LSTM and Transformer modules to better capture long-range dependencies in discrete audio tokens.
• SoundStorm enables non-autoregressive, parallel decoding using confidence-based strategies, achieving high-quality audio generation with reduced latency.
• Codec-based language models such as VALL-E, MusicGen, and AudioPaLM demonstrate strong zero-shot generalization across tasks like TTS, S2ST, and ASR using discrete tokens from EnCodec and SoundStream.
• Models like UniAudio and LauraGPT support a broad range of tasks—including speech enhancement, voice conversion, and text-to-music—by conditioning on diverse inputs like text, mel-spectrograms, and MIDI.
• The integration of discrete units from self-supervised models (e.g., Hubert, wav2vec 2.0) enables textless speech language modeling, with models like TWIST and pGSLM achieving strong performance in speech continuation and next-token prediction.

Figure 2: Codec-based Language Modeling